karpathy /neuraltalk2
Efficient Image Captioning code in Torch, runs on GPU
★ 5,603⑂ 1,267👁 267⑃ 22● 188▤ 0
+0
日增长
5,603
stars
1,267
forks
267
watchers
22
prs
188
issues
0
releases
README
NeuralTalk2 Update (September 22, 2016): The Google Brain team has released the image captioning model of Vinyals et al. (2015). The core model is very similar to NeuralTalk2 (a CNN followed by RNN), but the Google release should work significantly better as a result of better CNN, some tricks, and more careful engineering. Find it under im2txt repo in tensorflow. I'll leave this code base up for educational purposes and as a Torch implementation. Recurrent Neural Network captions your images. Now much faster and better than the original NeuralTalk. Compared to the original NeuralTalk this implementation is batched, uses Torch, runs on a GPU, and supports CNN finetuning. All of these together result in quite a large increase in training speed for the Language Model (~100x), but overall not as much because we also have to forward a VGGNet. However, overall very good models can be trained in 2-3 days, and they show a much better performance. This is an early code release that works great but is slightly hastily released and probably requires some code reading of inline comments (which I tried to be quite good with in general). I will be improving it over time but wanted to push the code out there because I promised it to too many people. This current code (and the pretrained model) gets ~0.9 CIDEr, which would place it around spot 8 on the codalab leaderboard. I will submit the actual result soon. !teaser results You can find a few more example results on the demo page. These results will improve a bit more once the last few bells and whistles are in place (e.g. beam search, ensembling, reranking). There's also a fun video by @kcimc, where he runs a neuraltalk2 pretrained model in real time on his laptop during a walk in Amsterdam. Requirements For evaluation only This code is written in Lua and requires Torch. If you're on Ubuntu, installing Torch in your home directory may look something like: bash $ curl -s https://raw.githubusercontent.com/torch/ezinstall/master/install-deps | bash $ git clone https://github.com/torch/distro.git ~/torch --recursive $ cd ~/torch; $ ./install.sh and enter "yes" at the end to modify your bashrc $ source ~/.bashrc See the Torch installation documentation for more details. After Torch is installed we need to get a few more packages using LuaRocks (which already came with the Torch install). In particular: bash $ luarocks install nn $ luarocks install nngraph $ luarocks install image We're also going to need the cjson library so that we can load/save json files. Follow their download link and then look under their section 2.4 for easy luarocks install. If you'd like to run on an NVIDIA GPU using CUDA (which you really, really want to if you're training a model, since we're using a VGGNet), you'll of course need a GPU, and you will have to install the CUDA Toolkit. Then get the cutorch and cunn packages: bash $ luarocks install cutorch $ luarocks install cunn If you'd like to use the c
Top issues
#32Bug on ARM hardware31 comments · 2015-11-30#64Am I the only one getting low CIDEr score with default setting?? I barely got 0.3~0.4 with 277500 iterations and what is CNN finetuning ?28 comments · 2016-01-29#154cant generate "coco/coco_raw.json" in neuraltalk221 comments · 2016-12-08#141Why the eval CIDEr: 0.673 is less than 0.915 comments · 2016-08-25#123Error: unable to locate HDF5 header file at hdf5.h12 comments · 2016-07-14#14Error when running eval.lua ("attempt to index field 'tensorOutput' (a nil value)")12 comments · 2015-11-26#149/torch/install/share/lua/5.1/hdf5/ffi.lua:56: ')' expected near '_close' at line 143611 comments · 2016-10-30#143Could neuraltalk2 caption images by using other language such as Chinese?7 comments · 2016-09-18