fix projection of refs in foreign collections

[WaVEV]

Handle refs from foreign collections.

[WaVEV]

Don't know if there could be a ref of refs. (it could, but Django creates it?)

[timgraham]

I think we've triaged pretty much all of the complicated queries by now, so if there's no test for it, I wouldn't worry about it.

[timgraham]

Can any of this be separate commits? It's a best practice not to mix non-behavior change refactors with bug fixes. For example, reclassifying test_ticket_24605 shouldn't be mixed in with other work. Edit: test categorization done in #100.

[timgraham]

My general rule is not to include blank lines when you have a comment separating a logical section. And if you feel like you need a blank line, add a comment instead.

[WaVEV]

Can any of this be separate commits? It's a best practice not to mix non-behavior change refactors with bug fixes. For example, reclassifying test_ticket_24605 shouldn't be mixed in with other work. Edit: test categorization done in #100.

This could be split into two commits or three if the last one is the features, I think. However, the refactor of project method and the fix for the incorrect references must go together.
The other part involves the management of references in the group by. All the code is related to the management of references in group by, project and the other stages, that implies changes in the flow of references. So, they are coupled enough to be in one PR, but we can split them into two commits.
EDIT: I decided to put only the changes that fix the expected failure test, the other changes belongs to aggregate_regress test suite.

[timgraham]

The --debug-sql output went from:

{'$project': {'_id': 1, 'isbn': 1, 'name': 1, 'pages': 1, 'rating': 1, 'price': 1, 'contact_id': 1, 'publisher_id': 1, 'pubdate': 1, 'store_name': '$name', 'annotations_store_books': 1, 'annotations_store': 1}}

to

{'$project': defaultdict(<CLASS 'dict'>, {'annotations_store': {'store_name': '$annotations_store.name'}, '_id': '$_id', 'isbn': '$isbn', 'name': '$name', 'pages': '$pages', 'rating': '$rating', 'price': '$price', 'contact_id': '$contact_id', 'publisher_id': '$publisher_id', 'pubdate': '$pubdate'})}.

1. Can we cast the defaultdict back to dict so it doesn't appear in the output like that? (The same issue affects $addFields.)

2. Is it worth leaving column references as 1 for brevity/simplicity or would this add too much complication to the code?

[WaVEV]

The --debug-sql output went from:

{'$project': {'_id': 1, 'isbn': 1, 'name': 1, 'pages': 1, 'rating': 1, 'price': 1, 'contact_id': 1, 'publisher_id': 1, 'pubdate': 1, 'store_name': '$name', 'annotations_store_books': 1, 'annotations_store': 1}}

to

{'$project': defaultdict(<CLASS 'dict'>, {'annotations_store': {'store_name': '$annotations_store.name'}, '_id': '$_id', 'isbn': '$isbn', 'name': '$name', 'pages': '$pages', 'rating': '$rating', 'price': '$price', 'contact_id': '$contact_id', 'publisher_id': '$publisher_id', 'pubdate': '$pubdate'})}.

1. Can we cast the defaultdict back to dict so it doesn't appear in the output like that? (The same issue affects $addFields.)
2. Is it worth leaving column references as 1 for brevity/simplicity or would this add too much complication to the code?

1. Yes, will do.
2. Also yes, we can just project if the name and the column's name are the same. I think it won't be so complicated.