Posted on General science, Research, Teaching

Best Machine Learning Resources

Machine learning is a rapidly evolving field that is generating an intense interest from a wide audience. So how can you get started?

For now, I’m going to assume that you already have the basic programming (ie general introduction to programming and experience with matrices) and mathematical skills (calculus and some probability and linear algebra).

These are the best current books on machine learning:

These are some out of date books that still contain some useful sections (for example, Murphy several times refers you to Bishop or MacKay for more details).

Here is a list of other potential resources:

 

Advertisement
Posted on General science, Research, Teaching

I3: International Institute for Intelligence

While I was previously discussing my opinion of Open AI, I mentioned that I would do something different if I was in charge. Here is my dream.

 

What OpenAI is Missing

Helping everyday people throughout the whole world.

OpenAI’s stated goal is:

OpenAI is a non-profit artificial intelligence research company. Our goal is to advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial return.

In the short term, we’re building on recent advances in AI research and working towards the next set of breakthroughs.

However, based on their actions so far, this interview with Ilya Sutskever, and popular press articles, the main focus of OpenAI appears to be advanced research in an artificial intelligence by stressing open source, as well as thinking longterm about the impacts of letting advanced artificial intelligence systems control large aspects of our life. While I strongly support these goals, in reality, these will not benefit all of humanity. Instead, it only benefits those with either the necessary training (which is a minimum of a bachelors, but usually means a masters or PhD) or money (to hire top people, buy the required computing resources, etc) to take advantage of the advanced research. So this leaves out the developing world as well as the poor in developed countries, ie contrary to their stated goal, OpenAI is missing the vast majority of humanity.

While one can argue that by making OpenAI’s research open source, eventually it will trickle down and help a wider swath of humanity. However, the current trend suggests that large corporations are best poised to benefit the most from the next revolution (I mean, who is more likely to invent a self driving car, Google, or someone in a developing country?). Additionally, these innovations focus on first world problems (since these are the highest paying customers). And finally, each round of innovation ends up creating fewer and fewer jobs (so the number of unemployed in developed countries may expand). I firmly believe that unless there is a global educational effort (and probably an implementation of basic income), the benefits of AI will be directed towards a tiny sliver of the world’s population.

 

My Proposal: I3

Here I lay out my proposal for a new institute that would actually expand the benefits of recent and future advances in machine learning / artificial intelligence to a wider swath of humanity. I don’t claim that it would truly benefit all of humanity (again, see basic income), but it is a way for research advances to reach a larger proportion of it.

I propose a new education and research institute focused on artificial intelligence, machine learning, and computational neuroscience which I’ll call the International Institute for Intelligence. I like alliterations, and since I think it should focus on three types of intelligence, I especially like the idea of calling it I3 or I-Cubed for short.

Why these three research areas? Well, machine learning is currently revolutionizing how companies use data and is facilitating new technological advances everyday. Designing artificial intelligence systems on top of these machine learning algorithms seems like a realistic possibility in the near future. The less conventional choice is computational neuroscience. I think it is important to include for two reasons. First, the brain is the best example we have of an intelligent system, so until we actually design an artificial intelligence, it seems best to understand and mimic the best example (this is the philosophy of Deep Mind according to Demis Hassabis). Second, the US Brain Initiative  and similar international efforts are injecting significant resources into neuroscience, with the hopes of sparking a revolution similar in spirit and magnitude to the widespread effect the Human Genome Project had on biotechnology and genomics. So I figure we might as well prepare everyone for this future.

So what would be the actual purpose of I3? Sticking with the theme of threes, I propose three initiatives that I will list in my order of importance as well as some bonus points.

 

1. International PhD Education

The central goal is to similar program to ICTP (International Centre for Theoretical Physics) but with a different research emphasis. So what is ICTP? It was founded by Nobel Prize Winner Abdus Salam and it has several programs to promote research in developing countries, including:

  • Predoctoral program – students get a 1 year course to prep them for PhDs
  • Visiting PhD program – students in a developing nation PhD program get to spend a couple of months each year for 3 years at ICTP to participate in their research
  • Conferences
  • Regional offices (currently Sao Paolo, Brazil, but more in the planning)

So the idea is to implement a similar program but with the research emphasis now focused on machine learning, artificial intelligence, and computational neuroscience. While I think the main thing is to get the predoctoral program and visiting PhD program started, eventually it would be great to have 5 regional offices spread throughout the developing world. For example, I think one is needed in South America (Lima, Peru?), one in Africa (Nairobi, Kenya?), and 2 in Asia (India, and China, but not in a traditional technological center). And assuming I3 is based in the US (see my case for San Diego below), it would be great to have an affiliate office in Europe, maybe in Trieste next to ICTP.

One additional initiative that I think could be useful would be paying people to not leave their country and instead help them establish a research center at their local universities. This could also wait until later because it might be easiest to convince some of the future alumni of the predoctoral or visiting PhD programs to return/stay in their home country.

A second additional initiative would be to encourage professors from developed and developing countries to take their sabbatical at I3. This would provide a fresh stream of mentors and set up potential future collaborations. This is a blend of two programs at KITP (this and that).

 

2. US Primary School Education

The science pipeline analogy is overused, but I don’t have a better one yet. So currently, the researchers in I3 focused areas are predominately male, white or Asian, and middle to upper class. So not a very representative sample of the US (or world) population. Therefore, the best longterm solution is to get a more diverse set of students interested in the research at a young age.

Technically this should have a higher priority over the next initiative (US College Education), but since there are other non-profits interested in this (for example, CodeNow), maybe I3 does not need to be a leader in this and instead can play a supporting role.

 

3. US College Education

And again back to science pipeline analogy, if we are to have a more diverse set of researchers, we need to encourage a diverse set of undergrads to pursue relevant majors and continue on into graduate programs. This won’t be solved by any single program, but here are some potential ideas.

  • US underrepresented students could apply for the same 1 year program that is offered to international students.
  • Assist universities in establishing bridge programs that partner research universities with colleges that have significant minority populations. A great example of this is the Vanderbilt-Fisk Physics program.
  • US colleges would also benefit from the proposed sabbatical program offered to international researchers. I also like the KITP idea of extending it to undergraduate only institutes (especially those with large minority populations) as a way to get more undergrads interested in research.
  • Establish a complete set of free college curriculum for machine learning, artificial intelligence, and computational neuroscience. While there are many useful MOOCs on these topics, I still don’t think they beat an actual course.

 

Bonus #1 : Research

ICTP has proven that it is possible to further global educational goals and still succeed at research. I would argue that the people working at I3 should mainly be evaluated for tenure based on their mentorship and teaching of students. Research of course will play a role (otherwise it would be poor mentorship of future researchers), but I think there shouldn’t be huge pressure to bring in grants, high-profile publications, etc. But even without that emphasis, there is no way that a group of smart people with motivated students will not lead to great research.

 

Bonus #2: International Primary and College Education

This is longer term, but if there are successful programs in improving the US primary and college education, international regional offices, and PhD alumni who are in their home countries, it seems like there should be possible to leverage those connections into a global initiative to improve primary and college education.

 

Final Thoughts

So Elon Musk, Peter Thiel, and friends, if you have another billion you want to donate (or Open AI funds to redirect), here is my proposal. In reality, implementing all of my ideas would probably cost several billions, but once you got the center founded, I think that it would be easy to get tech companies, the US government, and even UNESCO to help provide funding.

My final point is that I think San Diego would be a perfect location. I know I’m biased since I live here now, but there a many legitimate reasons San Diego is great for this institute.

  1. UCSD already partners with outside research institutes (Salk, Scripps, etc)
  2. UCSD (and Salk, etc) are leaders in all of these research areas
  3. It is extremely easy to convince people to take a sabbatical in San Diego

While there are many other great potential locations, I strongly suggest that I3 is not in the Bay Area, Seattle, Boston, or New York City. These cities already have plenty of tech jobs, please spread the wealth to other parts of the US.

Anyways, I’ll keep dreaming that someday I’ll get to work at a place like the one I just described.

 

Posted on Experts, Research

Deep Learning in Python

So maybe after reading some of my past posts, you are fired up to start programming a deep neural network in Python. How should you get started?

If you want to be able to run anything but the simplest neural networks on easy problems, you will find that since pure Python is an interpreted language, it is too slow. Does that mean we have to give up and write our own C++ code? Luckily GPUs and other programmers come to your rescue by offering between 5-100X speedup (I would estimate my average speedup at 10X, but it varies for specific tasks).

There are two main Python packages, Theano and TensorFlow, that are designed to let you write Python code that can either run on a CPU or a GPU. In essence, they are each their own mini-language with the following changes from standard Python:

  • Tensors (generalizations of matrices) are the primary variable type and treated as abstract mathematical objects (don’t need to specify actual values immediately).
  • Computational graphs are utilized to organize operations on the tensors.
  • When one wants to actually evaluate the graph on some data, it is stored in a shared variable that when possible gets sent to the GPU. This data is then processed by the graph (in place of the original tensor placeholders).
  • Automatic differentiation (ie it understands derivatives symbolically).
  • Built in numerical optimizations.

So to get started you will want to install either Theano (pip install theano), TensorFlow (details here), or both. I personally have only used Theano, but if Google keeps up the developmental progress of TensorFlow, I may end up switching to it.

At the end of the day, that means that if one wants to actually implement neural networks in Theano or TensorFlow, you essentially will learn another language. However, people have built various libraries that are abstractions on top of these mini-languages. Lasagne is one example that basically organizes Theano code so that you have to interact less with Theano, but you will still need to understand Theano. I initially started with Theano and Lasagne, but I am now a convert to Keras.

Instead, I advocate for Keras (pip install keras) for two major reasons:

  1.  High level abstraction. You can write standard Python code and get a deep neural network up and running very quickly.
  2. Back-end agnostic. Keras can run on either Theano or TensorFlow.

So it seems like a slam dunk right? Unfortunately life is never that simple, instead there are two catches:

  1. Mediocre documentation (using Numpy as a gold standard, or even comparing to Lasagne). You can get the standard things up and running based on theirs docs. But if you want to do anything advanced, you will find yourself looking into their source code on GitHub, which has some hidden, but useful, comments.
  2. Back-end agnostic. This means if you do want to introduce a modification to the back-end, and you want it to always work in Keras, you need to implement it in both Theano and TensorFlow. In practice this isn’t too bad since Keras has done a good job of implementing low-end operations.

Fortunately, the pros definitely outweigh the cons for Keras and I highly endorse it. Here are a few tips I have learned from my experience with Keras:

  • Become familiar with the Keras documentation.
  • I recommend only using the functional API which allows you to implement more complicated networks. The sequential API allows you to write simple models in fewer lines of code, but you lose flexibility (for example, you can’t access intermediate layers) and the code won’t generalize to complex models. So just embrace the functional API.
  • Explore the examples (here and here).
  • Check out the Keras GitHub.
  • Names for layers are optional keywords, but definitely use them! It will significantly help you when you are debugging.

Now start coding your own deep neural networks!

Posted on Everybody, Other

Basic Income

Recently the Swiss voted no on their referendum to implement basic income. Personally, I think that we should strongly consider implementing a basic income in the United States. At the minimum, I think that we deserve a national conversation on poverty that should include a serious discussion of the pros and cons of basic income. Therefore, I got really pissed off by this recent piece in the New York Times (or this, etc).

The author dismisses basic income out of hand for two major reasons:

  1. Cost (proposal of $10,000 to everyone over 21 for a total of $3 trillion)
  2. Negative effect on the poor (through government cuts due to the cost of implementing the above basic income)

I’ll go through the details below, but some rudimentary math shows that basic income could be paid for in the United States by tax increases that would not be a burden on the poor (or even most of the middle class). Thus, no government programs would need to be cut.

The purpose of this exercise is not to propose a foolproof implementation of basic income. Instead, I want to show that dismissing basic income due to cost is incorrect. If you want to debate basic income, the real issue is how our employment-centered economy would be changed by altering people’s motivation to work.

Here is a the conclusion of the calculations, please read on for all the details. I estimate that $11,500 (ie the US poverty line) could be paid to every non-Social Security receiving adult and $5750 (half the adult payment) to every child by adding a new flat tax on income (adjusted gross income) of 26.6% (see the appendix for alternative proposals that include a lower flat tax). This means that any individual that makes less than $49,500 would get MORE money from the government under this simple plan. Therefore, around 70% of non-Social Security US adults would get more money from the government.

 

Simple Calculation

Estimate Cost

I will start off by calculating the cost of basic income. First, how many people do we need to cover? I am going to ignore the 65 million that are on Social Security. My reasoning is that Social Security is almost a basic income (or could be with a few reforms) and that it is financially secure if we eliminate the cap ($118,500) on the payroll tax but do not increase benefits (see here and here for details).

Looking at the US census facts, there are 74 million children under 18, leaving 183 million US adults not on Social Security. I’m propose paying a half-adult benefit for children, so that means adult benefits will be paid to an effective population of 220 million.

Therefore, if each individual approximately gets the US poverty line, ($11,500), this would result in a total cost of $2.53 trillion.

Estimate Flat Tax

So how could we pay for this? The simplest possible mechanism would be a new flat tax on personal income.

The total US personal income in 2014 was $14.7 trillion. However, not all of that is taxable income (standard deductions, mortgage interest deduction, etc), so the actual taxable personal income is the adjusted gross income (AGI). Using some old numbers on AGI, I estimate that the total US AGI was $9.5 trillion in 2014.

Since the cost is $2.53 trillion, and US AGI is $9.5 trillion, that gives a flat tax rate of 26.6%.

Estimate Break Even Point

For a single individual, the standard deduction is $6300 (this amount of income is not taxed). It would take a taxable income of $43,180 to have a flat tax burden equal to the new basic income. Combine that with the standard deduction, and rounding a bit, leads to the conclusion that anyone making under $49,500 would gain money from the basic income/flat tax proposal.

 

Conclusion

Please don’t dismiss basic income purely out of cost. As the estimates above show, one could introduce basic income and pay for it with a new tax in a manner that preserves all other government programs.

I think there are two major reasons to embrace basic income:

  1. Fairness
  2. Needed security due to potential changes in employment

Maybe the fairness argument doesn’t fit with everyone’s political leanings, but I think the future of employment is strong motivation. Each new technological revolution seems to require fewer and fewer workers (compare Ford’s workforce vs Google’s). Since I don’t see that trend reversing and machine learning / artificial intelligence should actually accelerate it, I think we need to be proactive and provide a floor for people before we have large unemployment.

However, I recognize that basic income is a very controversial idea. That is why I am interested in seeing experimental implementations of it. While this experiment is nice, it really is too small to truly learn anything from. Instead, I would love to see a national trial. Why not start at a very small number, and slowly increase it over time? That would allow us to adapt to the changing culture (ie potentially NOT work centric) and make sure that there are no adverse incentives. For example, my dumb proposal of a half benefit to children needs to be more carefully monitored to ensure that people do not have children just for the sake of getting their share of the benefit.

No matter what you think, the debate isn’t going away. So we might as well start examining it now.

 

Appendix: Other Possible Tax Plans

Here I outline my own personal preferences for tax reforms (in addition to a flat tax) that could be used to pay for basic income. Note that all dollar amounts are per year.

Tax Reforms Within Current System

All numbers listed below are the estimated cost per year of the various deductions.

These are some tax reforms that many economists support:

I think this program is superseded by the introduction of a basic income:

And here are some additional reforms I support:

  • Tax capital gains and dividends as regular income ($85 billion)
  • Limit deductions for the wealthy ($25 billion)
  • Variety of corporate tax reforms ($40 billion) (I don’t understand depreciation so only the others on the list)

This comes to a total reform of $335 billion.

Note: I could have included food stamps or unemployment benefits in the superseded cost savings, but I’m going to assume that the benefits get reformed, but the money still is diverted towards health and employment initiatives respectively.

VAT

I would propose adding a federal value added tax as is common in Europe (see here for pros/cons). Bloomberg estimates that a VAT of 10% on a broad base of items would raise $750 billion per year. For ease of collection, this should be accompanied by a local/state replacement of the standard sales tax (which generates around $500 billion per year, or effectively a 6.66% VAT). I propose a 15% VAT (similar to European rates) that is split evenly between local/state governments and the federal government. This would generate $560 billion additional federal revenue (as a bonus the states get an additional $60 billion).

Financial Transaction Tax

This would impose a small fee on all financial transactions. If we implemented a 0.05% transaction fee (ie 50 cents on every $1000), this would raise an additional $90 billion.

Estate / Transfer Taxes

Currently $1.2 trillion is inherited per year, but estate taxes only bring in $8 billion in revenue. I suggest an estate / transfer tax reform to collect more revenue from this. I would structure it in a progressive manner (ie increasing with wealth), but I again just want to estimate the necessary average rate. If an average rate of 25% was applied to estates, this would lead to an additional $290 billion.

Personal Income Flat Tax

In the end, we still need $1.255 trillion in new revenue. And since the US AGI is $9.5 trillion (it would be slightly higher after the above reforms, but I will ignore that), that implies that we would need to implement a 13.2% flat tax to raise $1.255 trillion.

Summary

If one does a similar flat tax break even point, for people below $93,400 would receive more in basic income than in the flat tax. However, this is misleading since I no easy way to estimate the increase taxes due to the VAT. A worse-case scenario would be that people spend their complete income every year on VAT taxable items. Since there is currently an effective VAT of 6.66%, this is a VAT increase of 8.33%. So a break-even point for the combined flat tax / VAT rate (21.55%) would be $59,600. All the other taxes are much more complicated so I have no easy estimate for them.

The main point of this detailed appendix is that one could replace the flat income tax with a diverse set of taxes that again would not be an unfair burden on the poor or middle class. Additionally, the tax base would be diversified and less prone to swings in the economy.

Posted on General science, Research

General Programming Tips

I thought I would put together some useful programming tips that I have learned over the years. Most of these are general tips, but they are tailored towards Python.

  1. Zen of Python. Even if you don’t use Python, these are good ideas to internalize.
  2. The language documentation (Python’s standard library), StackOverflow, and Google searches are your best friends.
  3. Utilize modern IDEs (like Spyder for Python) and tab-completion to reduce the number of basic errors.
  4. Comments are not optional. The general logic of functions and objects should be understandable from the comments. Every block of code logic should have a short comment to aid future changes. If you find a chunk of code confusing now, it will be just as confusing if not worse in the future!
  5. Use sensible variable names. This cuts down on the number/length of comments.
  6. Try to adhere to the language standards (Python’s), but don’t obsess over it.
  7. Set your own consistent standards (Do variable names end in s or not? Do boolean variables have similar style names? Etc).
  8. When starting a project, do you best to get quickly get up to a basic working prototype. Working but incomplete code is always better than non-working code. Quick coding is aided by the next point…
  9. Outline your code before starting. My tips for outlining in Python are detailed after this list.
  10. Write modular code. Common tasks should be made into functions or objects.
  11. Avoid magic numbers and hard coded values. Better to include a set of named parameters in one section of your code where the basic logic of these variables is explained.
  12. Avoid multiple inheritance (check out this fun explanation of why this is bad).
  13. A program should have a standard interface, I like to call it main, and a way to run the standard interface with some default values. In Python, utilize if __name__ == ‘__main__’: to define standard parameters and then call main(parameters). This aids the goal of always having working code, as well as making it easier to interact with different programs.
  14. Check out these Python tricks (1-23 are the best, rest are more advanced).

 

Here are details on how I outline code in Python. I try my best to have running code at all times, even if it does absolutely nothing. If it isn’t real code, I leave it as a comment. Therefore, my programming tends to proceed as follows.

  1. Outline the general logic of the code in comments. Define needed functions, but at first have it take no actual variables (utilize pass to keep it as functioning Python code). In the comments inside a function, list the data type you think it should take in, what it should do, and what it should return.
  2. If you start to code a function or series of logic, you can safely leave it incomplete by having it raise NotImplementedError.
  3. Use assert to check any of your assumptions. A custom assert statement will save you lots of time later.
  4. While the Pythonic way is to utilize duck typing, I still prefer to do some type checking if there is potential for confusion. So I like to utilize things like isinstance or implement checks on attributes.
  5. Take advantage of your IDE’s additional formatting options. For example, Spyder specially highlights comments that start with TODO: with a little checkmark. Additionally, it supports code blocks and defines them by #%%. This lets you quickly run small chunks of a larger code.

What is the major advantages of coding like this?

  1. If your code always runs, it allows you to quickly find syntax errors and typos.
  2. You avoid implementing unused code. It sucks to really work on a code section to only realize later that you didn’t actually need it.
  3. You spend your time on the standard case and can add certain options or take care of edge cases when the appropriate time arises. Because sometimes that time will never arise…

Anyways, enough advice, start coding!

 

 

 

 

Posted on Everybody, General science, Research

Thoughts on OpenAI

OpenAI was started just over 6 months ago, and I feel like they have done enough to warrant a review of what they have done so far, and my thoughts of what they should do next.

What is OpenAI?

OpenAI was announced in December 2015 and their stated mission is:

OpenAI is a non-profit artificial intelligence research company. Our goal is to advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial return.

In the short term, we’re building on recent advances in AI research and working towards the next set of breakthroughs.

 

What have they done so far?

  1. Started a new, small (so far) research center
  2. Experimented with a novel organization of the research center
  3. Hired a variety of smart people
  4. Released a toolkit for reinforcement learning (RL)

Since it has only been six months and they are still getting setup, it is still difficult to assess how well they have done. But here are my first impressions of the above points.

  1. Always great to have more places hiring researchers!
  2. Way too early to assess. I’m always intrigued by experiments of new ways to organize research, since there are three dominant types of organizations today (academia, industry focused on development, and industry focused on longterm research).
  3. Bodes well for their future success.
  4. I have yet to use it, but the it looks awesome. Supervised learning was sped along by datasets such as UC Irvine’s Machine Learning Repository, MNIST, and Imagenet, and I think their toolkit could have a similar impact on RL.

 

What do I think they should do?

This blog post was motivated by me having a large list of things that I think OpenAI should be doing. After I started writing, I realized that many of the things on my wish list would probably be better run by a new research institute, which I will detail in a future post. So here, I focus on my research wish-list for OpenAI.

Keep the Data Flowing

As Neil Lawrence pointed out shortly after OpenAI’s launch, data is king. So I am very happy with OpenAI’s RL toolkit. I hope that they keep adding new datasets or environments that machine learners can use. Some future ideas include supporting new competitions (maybe in partnership with Kaggle?), partnering with organizations to open up their data, and introducing datasets for unsupervised learning.

Unsupervised Learning

But maybe I’m putting the cart (data) before the horse (algorithms and understanding). Unsupervised learning is tough for a series of interconnected issues:

  • What are good test cases / datasets for unsupervised learning?
  • How does one assess learning success?
  • Are our current algorithms even close to the “best”?

The reason supervised learning is easier is that algorithms require data with labels, there are lots of established metrics for evaluating success (for example, accuracy of label predictions), and we know for most metrics what is the best (100% correct label predictions). Reinforcement learning has some of that (data and a score), but is much less well defined that supervised learning.

So while I think the progress on reinforcement learning will definitely lead to new ideas for unsupervised learning, more work needs to be done directly on unsupervised learning. And since they have no profit motives or tenure pressure, I really hope OpenAI focuses on this extremely tough area.

Support Deep Learning Libraries

We currently have a very good problem: lots of deep learning libraries, to the point of almost being too many. A few years ago, everyone had to essentially code their own library, but now one can choose from Theano and TensorFlow for low end libraries, to Lasagne and Keras for high end libraries, just to name a few examples from Python.

I think that OpenAI could play a useful role in standardization and testing of libraries. While there are tons of great existing libraries, their documentation quality varies significantly, and in general is sub par (for example compared to NumPy). Additionally, besides choosing a language (I strongly advocate Python), one usually needs to choose a backend library (Theano vs TensorFlow), and then a high end library.

So my specific proposal for OpenAI is the following initiatives:

  1. Help establish some deep learning standards so people can verify the accuracy of a library and assess its quality and speed
  2. Set up some meetings between Theano, TensorFlow, and others to help standardize the backend (and include them in the settings of standards)
  3. Support initiatives for developers to improve documentation of their libraries
  4. Support projects that are agnostic to the backend (like Keras) and/or help other packages that are backend specific (like Lasagne) become backend agnostic

As a recent learning of deep learning, and someone who interacts extensively with non-machine learners, I think the above initiatives would allow a wider population of researchers to incorporate deep learning in their research.

Support Machine Learning Education

I believe this is the crucial area that OpenAI is missing, and it will prevent them from their stated mission to help all of humanity.

Check out a future post for my proposed solution…

Posted on General science, Research

Basic Bash

Basically these are the only things I know about Bash :). These aren’t all truly Bash commands, but instead these are common commands that everyone should know when using the Linux or Mac command line.

 

Notes

First, the command follows the $ and is listed in bold, the == is just a spacer, and everything else is a description of the command. The <> symbols designate where some other name should go there (like a file, folder, username, etc). The * symbol designates a wildcard, and can be used in conjunction with a partial search term. This means *.txt matches file.txt, file1.txt, etc.

 

Very Basics Commands

$ Ctrl + C  == Kill whatever is running in the foreground

$ tab == complete current typing

$ <command> –help == lists options of a given command

$ Ctrl + A == Go to the beginning of the line you are currently typing on

$ Ctrl + E = Go to the end of the line you are currently typing on

$ Ctrl + U == Clears the line before the cursor position. If you are at the end of the line, clears the entire line.

 

Where Am I and How to I Move Elsewhere?

$ pwd == print working directory

$ cd == change directory

$ cd / == go to root

$ cd .. == go up one level

$ ls == list files and folders

$ ls -a == list all files and folders (including hidden)

 

Basic File/Folder Manipulation

$ mkdir <folder name> == create new folder

$ cp <old file name>  <new file name> == copy and rename a file

$ mv <file> <folder> == move file to a folder

$ rm <file> == delete file

$ rm -r <folder> == delete folder

$ cat <file> == show content of file

$ head -n <number of lines> <file> == show top n lines of file

$ tail -n <number of lines> <file> == show last n lines of file

 

Nano

This is a basic file editor

$ nano <existing file> == opens up a file

$ nano <non-existing file> == creates and opens up a file

Within nano, the needed commands are listed at the bottom where the ^ symbol stands for Ctrl.

 

What is currently running? Can I stop it?

$ top == list all processes running on a computer

$ top -u <username> == lists processes being run by username

$ kill <pid> == kill a process identified by pid, which can be found by using top

$ killall -u <username> == kills all processes being run by username

 

How to run stuff in the background

$ <command> & == runs command in background

If you want to run jobs on a remote server, there will be some queueing system. Check out useful commands like qsub, qstat, etc. However, if you just want to run multiple processes on a single computer (even after logging off), Screen and Tmux are the tools you need. I personally use screen, and below are some useful commands.

 

Screen

$ screen -S <name> == new screen with name

$ Ctrl-a d == detach from screen

$ screen -r name == reattach to screen

$ screen -ls == list of screens

$ exit == kills currently attached screen

Example Running Code in the Background

Here is an example set of commands that will run a Python script in the background.

$ screen -S test

$ python test.py > out.txt 2>&1 &

$ Ctrl-a d

You now can safely exit and the computer will do all the tough work for you!

I did add one new command in there, the redirect (>). What this means is that Python runs test.py. If there is anything that should be printed to the terminal, it gets redirected and saved into the file out.txt. Otherwise when you reattach to the screen named test, you will just get the recent terminal output (usually there is some display length limit).

1 Comment Posted on General science, Research

Deep Learning: 0-60 in a few hours?

Here, I will try to outline the fastest possible path to go from zero understanding of deep learning to an understanding of the basic ideas. In a follow up post, I’ll outline some deep learning packages where you could actually implement these ideas.

I think by far the best introduction to deep learning is Michael Nielsen’s ebook. Before you get started with it, I think the minimum required mathematics includes an understanding of the following:

  • Vector and Matrix multiplication – especially when written in summation notation
  • Exponents and Logarithms
  • Derivatives and Partial Derivatives
  • Probability, mainly Bayes Theorem (not actually needed for Michael Nielsen’s book, but it is essential for later topics)

I really think that if you understand those mathematical topics, you can start reading the ebook.

Here is my proposed learning strategy. Iterate between reading the ebook (Chapters 1-5 only) and playing with this cool interactive neural network every time a new idea is mentioned. For a first pass, just read the ebook and don’t do the exercises or worry about actual code implementation. Additionally, chapter 6 introduces convolutional neural networks which are a more advanced topic that can be saved for later.

Once you have some intuition about neural networks, I recommend reading this review by several of the big names in deep learning. This will give you a flavor of the current status of the field.

Now you are ready to start coding!

PS. If you want to get into more advanced deep learning topics, check out my previous Deep Learning Unit. And to really get up to speed on research, there is a deep learning book that should be published soon.

 

Posted on General science, Research

Learning Python for Science

Here I outline how to learn Python on your own with emphasis on solving science problems. The first section applies to anyone, but the end is specialized towards computational problems that arise in science.

Python Basics

I recommend the following two tutorials:

Some additional resources that may be helpful include:

My suggested workflow:

  1. Do Codecademy and Python the Hard Way at the same time.
  2. If Codecademy/Python the Hard Way is too difficult, also read a Byte of Python.
  3. If Codecademy/Python the Hard Way is easy, use Think Python as an additional resource.
  4. If you are confused about a specific chunk of code, put it into Python Tutor which will walk you step by step through the program.
  5. Additionally, Google and Stack Overflow are extremely useful for coding questions or go to the original Python documentation.

The essential things one needs to learn about Python include:

  • data types: int, float, string
  • data structures: lists, dictionaries, tuples, sets
  • control statements: for, while, if else
  • print function
  • open / write to a text file
  • custom functions and objects
  • list comprehensions – comes up less often in numerical code, but still good to know

 

Numpy and Scipy

Numpy is the essential mathematics module in Python and is part of the larger Scipy project. All standard numerical needs are covered in Numpy, while more advanced functions are in Scipy.

I recommend the following tutorials:

  • Numpy’s tutorial
  • This fun tutorial that programs the Game of Life (GoL). I only recommend the section that implements GoL in Numpy, the rest of it is not essential. It also has a useful quick reference guide.
  • This Numpy tutorial from Scipy Lecture Notes

 

Matplotlib

Visualizing data is essential to understanding and communicating science ideas. Matplotlib is the standard plotting module. While it has its limitations, I still personally use it for my everyday plots. For more advanced plot types, check out Plotly, Seaborn, Mayavi, ggplot, and Bokeh.

And assuming you are new to making scientific figures, there are some good habits you should get into. First, read these tips from Plos. Second, never ever use rainbow colors aka jet. Color challenged people like myself will hate you. Please stick with a color map that uses shading sensible. Besides making me happy, it also is easier to print to gray scale.

 

Posted on Everybody, Research

Python on a Mac

I personally do most of my coding on my laptop, which is a Mac. Eventually that code gets run on a Linux server, but all initial coding, exploratory data analysis, etc is done on my laptop. And since I advocate for Python, I thought I would lay out all the steps I needed to do to setup my Mac in the easiest manner. (Note: probably similar steps on Windows, but I haven’t used a Windows computer in so long that I don’t know the potential differences).

8rYYWPNYNN-2

Unfortunately, the Python 2.x vs 3.x divide exists and so far, I have yet to be able to completely commit to 3.x due to a few packages with legacy issues. Luckily, there is a pretty easy solution below. Note, your Mac has Python preinstalled (go to terminal and type python to start coding…). However, if you want to update any packages, you can quickly run into issues. So it is easiest to install your own version of Python.

  1. Install Anaconda (I advocate version 2.7, Anaconda will call this environment root)
  2. I recommend using Anaconda Navigator and using Spyder for an IDE
  3. Install version 3.5 and make an environment (in Anaconda Navigator or terminal commands below): 
    $ conda create -n python3.5 python=3.5 anaconda
  4. You can switch between python environments  {root, python3.5} 
    $ source activate {insert environment name here}
  5. To add new python packages use conda or pip (anaconda has made its own pip the default)
  6. WARNING: always close Spyder before using conda update or pip. I got stuck in some weird place where Spyder would no longer launch. Apparently it can happen if Spyder is open and underlying packages get changed.

To get around the 2.x vs 3.x issue, go to your terminal and use pip install for the following packages: future, importlib, unittest2, and argparse. See the package’s website for details of any differences. Then, start your Python code with the following two lines:

from __future__ import (absolute_import, division, print_function, 
unicode_literals)

from builtins import *

For nearly all scientific computing applications, you are essentially writing Python 3 code. So make sure to read the correct documentation!

Personally, I found Anaconda to be a lifesaver. Otherwise, I got stuck in some weird infinite update loop to install all required packages for machine learning (specifically Theano).

Now you are ready to code! If you aren’t familiar with Python, my recommended tutorials will be in a future post.