One (not-entirely serious) possibility (with implicit spoilers), beneath the cut.
(gif via the comments thread at ComicBookMovie.com)
More seriously, I do wonder what Marvel's famed archer was up to during The Winter Soldier. I don't usually press the "where were all the other Avengers" questions in these movies, because I know it's the kind of nitpicky, detail-obsessed questioning that honestly can be kind of annoying (and besides which I don't really care that much). But in this case it does seem like a question worth asking, given Barton's role in SHIELD generally and close association with Natasha in particular.
I've heard speculation that he was on the trail of the Maximoff twins, which would make a certain amount of sense and allow an easy lead into Age of Ultron so that doesn't seem too unlikely.