Alessio Campanelli

Apache Lucene project develops open-source search software that provides indexing and searching functions. Go immediately to very simple code wich use only a small percent of complete capabilities. I Hope that this small article can stimulate curiosity for this great software.

A dataset analysed

My project is based on dataset of 4 Gb of tweets provide from my university. We use jsoup library for scraping all these txt files.

<o><t>1460847221990</t><l>en</l><p><![CDATA[{"filter_level":"low","retweeted":false,"in_reply_to_screen_name":null,"truncated":false,"lang":"en","in_reply_to_status_id_str":null,"id":721471655680151552,"in_reply_to_user_id_str":null,"timestamp_ms":"1460847221666","in_reply_to_status_id":null,"created_at":"Sat Apr 16 22:53:41 +0000 2016","favorite_count":0,"place":null,"coordinates":null,"text":"RT @qveensora: Sorry for making you feel like I need you","contributors":null,"retweeted_status":{"filter_level":"low","contributors":null,"text":"Sorry for making you feel like I need you","geo":null,"retweeted":false,"in_reply_to_screen_name":null,"truncated":false,"lang":"en","entities":{"symbols":[],"urls":[],"hashtags":[],"user_mentions":[]},"in_reply_to_status_id_str":null,"is_quote_status":false,"id":721471642795425792,"source":"<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone<\/a>","in_reply_to_user_id_str":null,"favorited":false,"in_reply_to_status_id":null,"retweet_count":1,"created_at":"Sat Apr 16 22:53:38 +0000 2016","in_reply_to_user_id":null,"favorite_count":0,"id_str":"721471642795425792","place":null,"user":{"location":null,"default_profile":false,"profile_background_tile":true,"statuses_count":56346,"lang":"en","profile_link_color":"0084B4","profile_banner_url":"https://pbs.twimg.com/profile_banners/612238377/1460771689","id":612238377,"following":null,"protected":false,"favourites_count":13535,"profile_text_color":"333333","verified":false,"description":"ig: qveensora","contributors_enabled":false,"profile_sidebar_border_color":"000000","name":"baja panti sora","profile_background_color":"C0DEED","created_at":"Tue Jun 19 01:17:00 +0000 2012","default_profile_image":false,"followers_count":5845,"profile_image_url_https":"https://pbs.twimg.com/profile_images/718854624833368064/cRJ-VPrj_normal.jpg","geo_enabled":false,"profile_background_image_url":"http://pbs.twimg.com/profile_background_images/378800000115632219/677754d26ed444e3c8f5500896e0d5e4.png","profile_background_image_url_https":"https://pbs.twimg.com/profile_background_images/378800000115632219/677754d26ed444e3c8f5500896e0d5e4.png","follow_request_sent":null,"url":null,"utc_offset":-14400,"time_zone":"Eastern Time (US & Canada)","notifications":null,"profile_use_background_image":true,"friends_count":3322,"profile_sidebar_fill_color":"DDEEF6","screen_name":"qveensora","id_str":"612238377","profile_image_url":"http://pbs.twimg.com/profile_images/718854624833368064/cRJ-VPrj_normal.jpg","listed_count":53,"is_translator":false},"coordinates":null},"geo":null,"entities":{"symbols":[],"urls":[],"hashtags":[],"user_mentions":[{"id":612238377,"name":"baja panti sora","indices":[3,13],"screen_name":"qveensora","id_str":"612238377"}]},"is_quote_status":false,"source":"<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Mobile Web (M5)<\/a>","favorited":false,"in_reply_to_user_id":null,"retweet_count":0,"id_str":"721471655680151552","user":{"location":"Palo Alto","default_profile":true,"profile_background_tile":false,"statuses_count":76,"lang":"en","profile_link_color":"2B7BB9","id":721471478399688704,"following":null,"protected":false,"favourites_count":0,"profile_text_color":"333333","verified":false,"description":"Buy Followérs and Likés for your Twitter, Facebook and Instagram http://bit.ly/1THk6v1#1756","contributors_enabled":false,"profile_sidebar_border_color":"C0DEED","name":"Buy Followėrs #1756","profile_background_color":"F5F8FA","created_at":"Sat Apr 16 22:52:59 +0000 2016","default_profile_image":true,"followers_count":0,"profile_image_url_https":"https://abs.twimg.com/sticky/default_profile_images/default_profile_2_normal.png","geo_enabled":false,"profile_background_image_url":"","profile_background_image_url_https":"","follow_request_sent":null,"url":null,"utc_offset":null,"time_zone":null,"notifications":null,"profile_use_background_image":true,"friends_count":0,"profile_sidebar_fill_color":"DDEEF6","screen_name":"AlexisCatt1","id_str":"721471478399688704","profile_image_url":"http://abs.twimg.com/sticky/default_profile_images/default_profile_2_normal.png","listed_count":0,"is_translator":false}}]]></p></o>

Creation of Lucene's docs

In this method, i read the local dataset and i create the lucene's docs for the query. It's important for me, sharing with you the importance of type property choosing. In fact StringField for the content of tweet NOT works at runtime query. Using TextField all kinds of query format works correctly

public static void createIndex() throws CorruptIndexException, LockObtainFailedException, IOException, TwitterException {
    
                    Directory indexDir = FSDirectory.open(new File(PATH_INDEXDIR));

IndexWriterConfig config = new IndexWriterConfig(Version.LATEST, new StandardAnalyzer());
                    IndexWriter indexWriter = new IndexWriter(indexDir, config);
                    
                    indexWriter.deleteAll();

List<Document> myLuceneList = new ArrayList<Document>();
                    
                    Document docLucene = new Document();     
                    
                    File dir = new File(PATH_DEBATES);
                    File[] files = dir.listFiles();

for(int i=0;i<files.length;i++){
                        currentFile = files[i];
                        currentContent = readContentFile(currentFile);

//scraping of single object of dataset of twitter
                        org.jsoup.nodes.Document docJsoup = Jsoup.parse(currentContent);
                        Elements elementsP = docJsoup.select("p");             
                        Elements elementsT = docJsoup.select("t");             
                        Elements elementsL = docJsoup.select("l");  
                    
                        //scan of tag 'p' that contain the real value of single tweet 
                        for (int i = 0; i < elementsP.Size(); i++) {
                            String jsonContent = elementsP.get(i).text();           
                            Status status = TwitterObjectFactory.createStatus(jsonContent);

docLucene = new Document();
                            docLucene.add(new StringField("idTweet", elementsT.get(i).text(),Field.Store.YES));
                            docLucene.add(new StringField("languageTweet", elementsL.get(i).text(),Field.Store.YES));
                            docLucene.add(new StringField("tweetUser", status().getUser().getName().toString().toLowerCase(),Field.Store.YES));
                            docLucene.add(new TextField("tweetText", status.getText().toString().toLowerCase(),Field.Store.YES ));
                        
                            myLuceneList.add(docLucene);
                        }
                    }
                  }

Implement search Engine

It's intersting that object ScoreDoc contain a lot of attributes utils for analysis. In my case i use rating. The keys 'tweetUser' and 'tweetText' are the keys choose previously for Lucene.