SOLR: Non English (Latin) Characters in Field Name

The SOLR documentation mentions following requirement for defining name of a field.

name

The name of the field. Field names should consist of alphanumeric or underscore characters only and not start with a digit. . . .

While working on a dictionary website, the JSON documents that I created had field names in Hindi. After indexing the data I was surprised to see that field names in the data were converted to multiple underscore letters e.g. field name शब्द was converted to ____. According to SOLR documentation शब्द should have been allowed as field name.

Looks like SOLR developers have assumed that only 26 letters in Latin script are alphabets. Mentioning this assumption explicitly in documentation would have been helpful.

After a closer scrutiny of solrconfig.xml file, I found following configuration, which converts anything that is not Latin alphanumeric in field name to underscore while indexing the data.

<updateProcessor class="solr.FieldNameMutatingUpdateProcessorFactory" name="field-name-mutating">

   <str name="pattern">[^\w-\.]</str>

   <str name="replacement">_</str>

</updateProcessor>

Changing the pattern regex for FieldNameMutatingUpdateProcessorFactory to something like below will allow SOLR to accept non Latin alphabets in field name.

   <str name="pattern">[\s]</str>

[\s] is a too generic pattern to use in real life scenario. This pattern should be further restricted to a limited set of characters that one intend to use in field name.

How to customize Moodle Mobile App?

Chinese Virus (COVID-19) has created unimaginable chaos that no one could have thought about. As a side effect most of us have forced spare time to share with family at home. I utilized a part of this time to explore something new and that is Moodle.

Moodle is a widely popular Learning Management System and it has a Mobile app to deliver the content to complement the web deliver channel. Installation of Moodle and setting up few courses was a fairly simple activity. Sample courses were rendering well on standard Moodle App for Android.

Then I decided customize the mobile app to give it a unique branding. I was expecting it to be a simple task but it came out to be a tough activity. My lack of previous experience in mobile app development was a big contributor to that. Moodle for Mobile forum was not of much help. After putting lots of effort I was able to get it done. I have documented the steps required to build and customize the Moodle App.

Environment Setup

Moodle has a good document to setup the local development environment. Though it is bit outdated at places but following that I was able to setup the development environment on my Mac.

Now clone the source code from https://github.com/moodlehq/moodleapp

Configuration

Next step is to change the configuration to give it a custom branding. Here is the list of files and the relevant configuration properties that needs to be changed.

• Change following properties in config.xml.

• Widget id

• Widget version

• App name

• Description

• Author

• AppendUserAgent

• URL_SCHEME (change it only if plan to block official Moodle App from accessing your website)

• Update server side property “tool_mobile | forcedurlscheme” at the above-mentioned URL.

• Make following changes in src/config.json

• app_id

• appname

• desktopname

• versioncode

• versionname

• Empty demo_sites element

• customurlscheme (related to URL_SCHEME)

• siteurl

• sitename

• privacypolicy

• Update custom url scheme in package.json (related to URL_SCHEME)

Logo & Splash Screen

Having own logo and splash screen is what people want in custom moodle app. Below is the list of images for logo and splash screen that you have to put in source code.

• resources/icon.png

• resources/splash.png

• resources/icon/icon-background.png

• resources/icon/icon-foreground.png

Following images should be removed, so that above-mentioned images are used during build process.

• resources/splash.png.md5

• resources/android/icon.png

• resources/android/icon.png.md5

• resources/android/icon-foreground.svg

• resources/ios/icon.png

• resources/ios/icon.png.md5

Now run ionic cordova resources command to generate icons and splash images for various sized screens. Successful execution of this command will print logs as shown below.

IMUL-ML0245:moodleapp vinodsingh$ ionic cordova resources

> cordova-res

[cordova-res] Generated 24 resources for android

[cordova-res] Generated 50 resources for ios

[cordova-res] Wrote to config.xml

  ──────────────────────────────────────

     Update available: 5.2.3 → 5.4.16

       Run npm i -g ionic to update

  ──────────────────────────────────────

Note:- In my case it did not created 4 android icons whose name were ending in *-smallicon.png. I removed those images and deleted corresponding entries from config.xml.

Very informative documentation about creating icon and splash images is available here -

https://ionicframework.com/docs/cli/commands/cordova-resources

https://ionicframework.com/blog/automating-icons-and-splash-screens/

Firebase Project

For every Android app a Firebase project is madatory. Go to Firebase Console and create a new project and add the app to that. Download the google-services.json file and place it at the root folder.

Build - Debug

Now all required changes are done and we are ready to build a debug version of the app. Run ionic cordova build android to build the app with debug messages. It may fail with the following error.

Using cordova-fetch for cordova-android@8.0.0

Adding android project...

Creating Cordova project for the Android platform:

Path: platforms/android

Package: bharat.sarathi.mobile

Name: _____.____

Activity: MainActivity

Android target: android-28

Subproject Path: CordovaLib

Subproject Path: app

Android project created with cordova-android@8.0.0

Unable to graft xml at selector "/manifest/uses-feature[@android:name='android.hardware.location.gps']" from "/Users/vinodsingh/code/GitHub/moodleapp/platforms/android/app/src/main/AndroidManifest.xml" during config install

[ERROR] An error occurred while running subprocess cordova.

        

        cordova platform add android --save exited with exit code 1.

        

        Re-running this command with the --verbose flag may provide more information.

Running same command with --verbose provides precise error message as shown below.

[12:31:35] Local gulp not found in ~/code/GitHub/moodleapp

[12:31:35] Try running: npm install gulp

After installing local gulp reran the ionic cordova build android --verbose command and it succeeded. The APK file will be placed here - platforms/android/app/build/outputs/apk/debug/app-debug.apk. Debug app can be tested on an android emulator.

Running the build command again may fail with following error -

Users/vinodsingh/code/GitHub/moodleapp/platforms/android/app/src/main/AndroidManifest.xml:33:5-66 Error:

Element uses-feature#android.hardware.location.gps at AndroidManifest.xml:33:5-66 duplicated with element declared at AndroidManifest.xml:32:5-91

/Users/vinodsingh/code/GitHub/moodleapp/platforms/android/app/src/main/AndroidManifest.xml Error:

Validation failed, exiting

To fix this error remove the below line from AndroidManifest.xml file and run the build command again.

    <uses-feature android:name="android.hardware.location.gps" android:required="false" />

Intent Filters

Testing revealed an issue with the app. After authentication in browser control does not go back to the app. User is stuck in the browser itself. This issue makes app unusable. After lots of struggle, I figured out that the following intent-filters are missing from main activity in AndroidManifest.xml.

    <intent-filter>

        <action android:name="android.intent.action.VIEW" />

        <category android:name="android.intent.category.DEFAULT" />

        <category android:name="android.intent.category.BROWSABLE" />

        <data android:scheme="URL_SCHEME" />

    </intent-filter>

    <intent-filter>

        <action android:name="android.intent.action.VIEW" />

        <category android:name="android.intent.category.DEFAULT" />

        <category android:name="android.intent.category.BROWSABLE" />

        <data android:host=" " android:pathPrefix="/" android:scheme=" " />

    </intent-filter>

Ideally they should have been there due to the inclusion of customurl-plugin. Adding these intent-filters manually fixed the issue.

Build - Release

After successful testing the app on emulator, I am ready for production build and test on a real device. Reading ionic publishing document is recommended before proceeding with next steps.

Create key [one time]

We have to create a key for signing the app. The Java keytool command as shown below can generate the required key.

keytool -genkey -v -keystore my-release-key.keystore -alias alias-name -keyalg RSA -keysize 2048 -validity 10000

Build

Moodle suggests few edits to the source code to make it production ready. Suggested code changes are documented here. After making code changes run the build command to create APK without debug messages.

ionic cordova build android --prod --release

It will create a apk file at location - platforms/android/app/build/outputs/apk/release/app-release-unsigned.apk location.

Sign the APK

Now this release apk file needs to be signed with key that we generated a while back.

jarsigner -verbose -sigalg SHA1withRSA -digestalg SHA1 -keystore my-release-key.keystore app-release-unsigned.apk alias-name

Next is to zip align the APK using zipalign tool available in Android SDK.

~/Library/Android/sdk/build-tools/29.0.3/zipalign -v 4 app-release-unsigned.apk my-production-ready.apk

Now APK is similar to what one will download from the Play Store. It is ready for testing on an emulator or real Android device.

Why NoSQL is in fashion?

Couple of years back NoSQL was relatively unheard term. It has become a new buzzword in town, catching people's imagination. Almost all big daddies of Internet (Google, Facebook, Yahoo, Twitter, Amazon, Linked-in etc.) are using NoSQL at massive scale and many of them have their own NoSQL products. Every month there is a launch of new NoSQL product catering to some specific use cases / requirements. So what are the reasons behind NoSQL's rise to popularity?

Background

RDBMS and SQL has ruled the world for last 3 decades. They were (are) the defacto standard for storing business data. In fact there was no alternative to RDBMS and SQL combination. When RDBMS were developed the disk space was scarce, hence a lot of thought has gone around saving precious disk space. Development in hardware technologies have made available literally unlimited storage space at lower prices. Evolution of Internet in last decade has paved way for altogether different kind of applications and use cases, which were never thought earlier. Web2 applications (say social networking) have entirely different requirements than typical enterprise applications. Other than that Internet has changed the way application developers think about the data and its usage & storage.

Requirement

Nowadays enormous amount of data is being produced everyday and the rate at which data is produced is also increasing. Data size is so huge that organizations spread it across data center as it does not fit in a single rack or data center. Though data availability is also an equally important factor in using multiple data centers for storage. Applications have to available 24x7 no matter what. Data has to survive the data center failure as well, forget about failure of a disk. Instead of archiving data, Facebook type of applications keep it online forever.

Social networking sites are coming up with new features quite frequently and sometimes rolling back them. At times their new features were never thought about earlier. They don't want to be confined by the schema constraints of RDBMS which is quite difficult (though not impossible) to change. Support for evolving schema has become a necessity and NoSQL solutions are a right fit there instead of RDBMS.

Data size, flexible schema and availability are the 3 key factors where traditional RDBMS do not perform well after a certain limit.

Thought Process

Web2 applications has changed how people think about application architecture. It is an altogether different thought process then what it was few years back.

While designing an enterprise application we used to think about "how to store the data" but today the primary thought is "how we will use the data". This is a paradigm shift. Data storage is designed based on how that is going to be used, leading to optimized performance for a given requirement at times at the cost of flexibility.

"Referencing" entities was a wonderful feature in RDBMS but for large scale web 2 application that does not matter at all. For them instead of referencing "embedding" is the killer feature. Storing related data (say personal detail and address) together makes sense as they don't have any meaning independently. This is similar to denormalization in RDBMS.

For many features of web 2 applications ACID transactions are also not a primary requirement. Say if a tweet does not gets published many users won't mind to type that again and hit the enter button as long as that does not happen frequently. If avoiding ACID transactions increases the throughput many fold for not so critical features, what else you want :-). It is similar to old MySQL though with much larger data-set than what MySQL can support.

Conclusion

There are several use cases where NoSQL perform much better and it makes perfect sense to use NoSQL instead of trying to scale the RDBMS by using expensive methods. RDBMS’s are really good at doing what they do, which is storing flat, relational, tabular data in a consistent manner and getting wonderful reports out of that data. They still remain the best solution for storing relational data.While NoSQL databases are good at performance, availability, schema-less persistent etc., which has become a basic requirement for today's applications. Support for these features is the main reason for meteoric rise to fame of the NoSQL.