Establish a build/exec/deploy system

[Martii]

Fork a command line minifier for dev tools on the OpenUserJS account

This will be needed in a while to minify what we want to before deploying along with a deploy.sh that is kept with sizzle and maybe here... depends on the secrets that need to be kept.

Dev and GH source should remain unminified at all times... only time minified should be existing is on deployment/testing... this may require some extra time with making sure minification doesn't clobber anything... usually they are quite good at what they do by now but to be safe an extra testing measure should be done.

This is probably a long term issue since it's going to take some time to undo what's been done.

---

We could also use, instead, a shell script with git directly to a folder and then use some .sh scripts to move things to the appropriate locations... keeping in mind .gitignore . e.g. we really need a build.sh system and exec.sh type system.

---

This section may change constantly...

• VPS
• Initialization system... perpetual tweaking possibly. Invoked via $ npm install or $ node dev/init.js
  • Look into https://github.com/alexanderGugel/ied as alternative
• Exec... Invoked via $ npm start or $ node app.js ... process management roughed in.
• Deploy... $ git pull on VPS
• Minify CSS on the fly... partial... needs enabling of cache system and placement on the VPS tree with automated cleanup
• Minify JavaScript on the fly... partial... needs enabling of cache system and placement on the VPS tree with automated cleanup... wants Test and support new option for *express-minify* with keeping comments #432
• LESS caching (maybe... probable that express-minify is already doing this. Also bootstrap@4.x will use SASS)
• Minify HTML
• See if nock is viable for testing requests that aren't normally available in dev
• ...

[cletusc]

This is where something like #97 Grunt or gulp.js would come in. As far as I'm concerned, all tools (including deploy scripts) should be in the repo in some sort of build or deploy folder to store temp files (auth info or other files). Either build tool can concatenate and minify JS, CSS, or precompile mustache templates.

[Martii]

Thanks Cletus... I'll start reading up on these, including grunts repo. We definitely seem to have calls over many CDNs. This really should stop and be centralized. A build system could manage some of this... hopefully without merge conflicts... but I see multiple .css/.js files from different CDNs with the same naming. While I'm sure those sites might be credible it's impending doom if they change something and kill OUJS for some reason.

[cletusc]

We can also use something along the lines of Bower in conjuction with these. You can set it up to pull a specific URL or repo, e.g. a known working version from a CDN or github, and then the build system can pull from that (just like anything in node_modules from npm install ...). When we need to update a version, it's as simple as changing the Bower info, then testing again. For this particular site, Grunt with Bower is probably the ideal combo.

[joeytwiddle]

CloudFlare seems to be a pretty popular CDN: https://www.cloudflare.com/features-cdn

They have this collection for rarer scripts: http://cdnjs.com/

At least if they go down, many other websites will be broken too. ;)

Obviously Google has their own CDN: https://developers.google.com/speed/libraries/devguide

Some advantages of using a CDN over self-hosting:

• Faster load for new users, because they may already have the scripts (URLs) in their cache (from visits to previous websites using the CDN).
• Faster load because the CDNs set huge expiry dates (so we don't have to fiddle with that).

Disadvantages:

• If the CDN is unreachable by the user for whatever reason, it affects our site. If we self-host all the files, then we are self-contained.
• Hosting the scripts ourselves might work better for tor users (not sure about that).

[Martii]

I understand the only perk is for speed but with recent experiences these CDNs are more of a pain than a plus.

Disadvantages of a non-owned/managed CDN	Advantages of a non-owned/managed CDN
Lack of consistency in the code base. e.g. we are referencing CSS files that have the same name and are just slightly different but presumably the same purpose... this is bad.	Occasional and intermittent speed improvement
Lack of security esp. with .js hosted files
Lack of privacy. CDNs have always had this issue
On every machine in my nets the cache is cleared on browser close and open... occasionally during if it's initiated. Setting expiry dates isn't that difficult provided web standards are adhered to. ;)
Cloudflare is where we currently get our icons from and sometimes they never show up.
Ace editor has cross-domain issues when used on dev and occasionally on pro.
Their site goes down... we could be foo'd
Add-on Incompatibility of known blacklists

In general it's a bad practice if we don't own the management rights to the hosting space without some control of versioning. The weight of this merits hosting things ourselves or implementing project build controls. The files are relatively small so I don't think it's going to be much of an issue. The solution currently is to put these on our controlled site and have some sort of management to keep them up to date. I'd much rather we kill the site accidentally than someone else out there.

[Martii]

@cletusc
Hmm grunt requires a CLA to be signed if we ever need to contribute upstream. This is probably not a wise thing to mix with GPL... GPL will always take precedence in our project and usurping that clearly defined authority could be deemed as unsupported. There is also GHs licensing which this can conflict with. I'll still look into it some more but the gulp repo might be more viable but don't know if it's Bower compatible. Do you know off hand? Also if I was grunt I wouldn't be advertising that Adobe is one of their users... imeo they have some of most buggy closed source software around and we all get to pay for it usually. ;)

---

There's still always the option of doing this as natively as possible with git. Bash shell scripts can do just about anything and has a longer mtbf since it's been around since the dawn of terminal shells.

---

Related chatter:

• Add the bootswatch fork to OpenUserJS Organization #223

[Martii]

@cletusc Cc: @sizzlemctwizzle
Silly question... why would we want to use Bower which has a wrapped package for their system instead of $ npm directly which supports GH repo's? So far jQuery and ACE are on Bower but they are also on GH (jQuery on npm however ACE is not that I can find but I searched on GH for #281)... so couldn't we just use the npm package.json to do the downloading/updating/versioning? Does nodejitsu support GH downloads in package.json? We'll still need something to minify and I'm liking the docs for gulp since it doesn't use intermediate files as much. I assume that we can pull from the node_modules folder anything we want to a build or test folder, and deploy folder. (and/or reuse the dev folder)

$ npm install -h
npm install
npm install <pkg>
npm install <pkg>@<tag>
npm install <pkg>@<version>
npm install <pkg>@<version range>
npm install <folder>
npm install <tarball file>
npm install <tarball url>
npm install <git:// url>
npm install <github username>/<github project>

[cletusc]

so couldn't we just use the npm package.json to do the downloading/updating/versioning?

We could; the only downside is we wouldn't be able to do single files as with Bower.

I assume that we can pull from the node_modules folder

Or serve directly from the node_modules folder, if we ever wanted to go that route, but yes, npm/bower basically just downloads the package to node_modules/bower_components and you can deal with it just like any other file/folder.