Cannot search for KVs or tags

[allanlasser]

Challenge

Right now, there's no way to provide users with an index of tags/keys/values they've used (or explore tags across DocumentCloud independent of documents). If we want to provide tags as a search facet, we'll need a way to search against them.

Proposed solution

One way to implement this would be to add an endpoint that supports searching against keys and their values.

• We could then use that endpoint from the frontend to present a tag search UI that searches KVs with key=_tag&value={x}; would expect the results to conform to the type Array<{key: "_tag"; value: string[]}>.
• We could allow users to search KVs with key={x}; would expect the results to conform to the type Array<{key: string; value: string[]}>.
• We could allow users to find a specific KV pair by value, like key={x}&value={y}; would expect the results to conform to the type Array<{key: string; value: string[]}>.
• Filter by project, so I can get a list of available keys for a single project
• Implicitly limit to documents a user can see

[duckduckgrayduck]

When I thought of search, my brain at first went to Solr. So I did some digging through the Solr documentation. We can right now do a Solr search for the values of a given key using facet fields, which are built into dynamic fields that are indexed (data_* is already).

For example something like the following can be run if I want to find all of the possible values for the key color and the frequency of each value.

s@s:~/dev/documentcloud-frontend$ curl "http://localhost:8983/solr/documentcloud/select?q=*:*&facet=on&facet.field=data_color&rows=0&wt=json"
{
"responseHeader":{
"status":0,
"QTime":1,
"params":{
"q":":",
"facet.field":"data_color",
"rows":"0",
"facet":"on",
"wt":"json"}},
"response":{"numFound":59,"start":0,"docs":[]
},
"facet_counts":{
"facet_queries":{},
"facet_fields":{
"data_color":[
"blue",2,
"red",1]},
"facet_ranges":{},
"facet_intervals":{},
"facet_heatmaps":{}}}

This doesn't however allow us to search for possible keys (someone adding a new key/value pair could spell color "colour" for example and there wouldn't be a way to tell doing this approach without creating some other kind of store that contains all the possible key names for a given set of documents.

We could make the value search more performant by adding docValues=True to the data_* dynamic field as described in Solr documentation.

After consulting with Mitch, I am going to try do this with Django queries instead, and make it more performant by keeping the query size smaller (like only enabling it within a project to start)