Let us assume your custom language is called 'lang1'.
1. Create mode and worker content files with the naming convention a 'lang1.js' and 'lang1_worker.js.
2. Download ace source code from https://github.com/ajaxorg/ace.
3. In ace source code go to '/lib/ace/mode' folder and place your lang1.js and lang1_worker.js files.
4. Build ace source code with command ' node ./Makefile.dryice.js '.(Check github page for instructions)
5. Once successfully built, go to 'build/src' folder and you can find your mode file as 'lang1.js' and your worker file created as 'worker-nel.js'.
Now you can place these files in your source code location and work with them. Do remember to update the worker name to new one, when creating the worker client.
var worker = new WorkerClient(["lang1"], "lang1/mode/worker-lang1", "Lang1Module");