Messages

16

Software Engineering / WebGL available in Firefox Nightly

« on: April 20, 2017, 03:04:50 PM »

We mentioned that WebGL had landed in WebKit source, when it joined Firefox.

Vladimir Vuki?evi? of Mozilla has posted on how it shows up in a nightly instead of just source (which requires a compiler flag etc.

This is incredibly exciting, as Jon Tirsen said:

Your next 3D shooter will sport a nice “Your browser is not supported please install Chrome, Safari or Firefox.” (Re: WebGL.)

Hopefully IE gets there too of course (Opera is in the group so we should see something there too).

Here is Vlad:

Along with the Firefox implementation, a WebGL implementation landed in WebKit fairly recently.  All of these implementations are going to have some interoperability issues for the next little while, as the spec is still in flux and we’re tracking it at different rates, but will hopefully start to stabilize over the next few months.

If you’d like to experiment with WebGL with a trunk nightly build (starting from Friday, September 18th), all you have to do is flip a pref: load about:config, search for “webgl“, and double-click “webgl.enabled_for_all_sites” to change the value from false to true.  You’ll currently have the most luck on MacOS X machines or Windows machines with up-to-date OpenGL drivers.

We still have some ways to go, as there are issues in shader security and portability, not to mention figuring out what to do on platforms where OpenGL is not available.  (The latter is an interesting problem; we’re trying to ensure that the API can be implementable on top of a non-GL native 3D API, such as Direct3D, so that might be one option.)  But progress is being quickly made.

When paired with high-performance JavaScript, such as what we’ve seen come from both Firefox and other browsers, should allow for some exciting fully 3D-enabled web applications.  We’ll have some simple demos linked for you soon, both here

17

Software Engineering / Re: Overlap Examination System

« on: April 20, 2017, 02:59:49 PM »

Thanks for letting us know madam..

18

Software Engineering / Making the OpenSocial API feel more at home

« on: April 20, 2017, 02:58:09 PM »

Chris Chabot has been doing a lot of experimentation with the new OpenSocial APIs. He has written up his experience and created two prototype wrappers.

The first short article has some general information and background.

The second article includes the first library you can tell to load (owner, viewer, ownerFriends and/or
viewerFriends) information, and presents this information in an uniform way (instead of having to do different type of calls for different information fields) and with proper consistent error handling. With it you can very easily create your first OpenSocial container application in a friendly prototype style environment. You can take a direct look at the library itself.

The third article contains a Ajax.Request implementation, since Prototype’s version won’t work well or even at all in the cross domain environment of open social containers, it allows you to re-use your current Prototype based programs by trying to mimic Prototype’s Ajax call as well as possible given the constraints of the situation. Under the hood, _IG_FetchContent is used to talk back to the server.

It is good to see people take the raw APIs and make them feel more like their library of choice.

19

Software Engineering / Canvas Color Cycling, digital imaging

« on: April 20, 2017, 02:54:04 PM »

Interest in Canvas, as well as mobile apps, has led to a renaissance of old-school 8-bit graphics. Joe Huckaby of Effect Games has been playing around with color cycling, leading to some stunning effects.

Anyone remember Color cycling from the 90s? This was a technology often used in 8-bit video games of the era, to achieve interesting visual effects by cycling (shifting) the color palette. Back then video cards could only render 256 colors at a time, so a palette of selected colors was used. But the programmer could change this palette at will, and all the onscreen colors would instantly change to match. It was fast, and took virtually no memory.

There’s a neat optimization going on here too: instead of clearing and redrawing the entire scene with each frame, he only updates the pixels that change:

In order to achieve fast frame rates in the browser, I had to get a little crazy in the engine implementation. Rendering a 640×480 indexed image on a 32-bit RGB canvas means walking through and drawing 307,200 pixels per frame, in JavaScript. That’s a very big array to traverse, and some browsers just couldn’t keep up. To overcome this, I pre-process the images when they are first loaded, and grab the pixels that reference colors which are animated (i.e. are part of cycling sets in the palette). Those pixel X/Y offsets are stored in a separate, smaller array, and thus only the pixels that change are refreshed onscreen. This optimization trick works so well, that the thing actually runs at a pretty decent speed on my iPhone 3GS and iPad!
 

20

Software Engineering / Speech Recognition with Javascript

« on: April 20, 2017, 02:49:35 PM »

Recently Google’s free text to speech api has made the rounds. The reverse is also possible, converting speech to text.

With speechapi.com’s javascript API, it is possible to build interesting speech-web mashups that include both speech-to-text as well as text-to-speech.

A combination of several technologies and open source tools make this possible. In the browser, Flash is used to access the microphone and stream the audio to an RTMP server. Red5 is used because its a versatile media server that has the benefit of being open source and free.

Once that audio is received on the server, it needs to be converted to text. There are many speech recognition engines to choose from. Many are proprietary and provide very good accuracy results but they are pricey and closed source. There are some state of the art opensource speech recognition engines too, such as julius and Sphinx to name a couple. The speechapi service uses sphinx because it is license friendly and has a strong community.

Now this is great, we can transmit audio and convert it to text but we need to control the process and use the results in the web page. That is where Javascript comes in. Speechapi.com provides a Javascript API. There is a setupRecognition method that sets up the grammar used in the speech-to-text process. There is a simple grammar mode, where you can just provide a comma seperated list of words. JSGF is also supported and is useful for more complex grammars. There are also methods that communicate with the flash control to indicate when to start transmitting audio and when to stop transmitting audio. You can also use the flash controls built in press to speak button to specify the speech endpoints.

Recognition results are returned to your web page in a callback that you specify in the speechapi constructor. The results are passed from the server to client as a JSON string. The result object contains the raw text results as well as other information that can be useful for you speech client, like pronunciation and “grammar tags” that can be useful for semantic interpretation of the results.

We think this technology is pretty cool and we encourage you to try it out. You can try it for free at speechapi.com where you just include a few lines of of javascript and html into your webpage to enable speech recognition. We are also open sourcing the package over the next few months, so sign up at our site if your interested.
Thanks for your interest
Younus

21

Software Engineering / SoundManager2 now with HTML5 Audio

« on: April 20, 2017, 02:46:47 PM »

Scott Schiller, the best moustache-d frontend engineer around, has updated his awesome SoundManager library. The latest SoundManager 2 version now comes with free HTML5 Audio support which makes it a HTML5 Audio()-capable JavaScript Sound API, backwards-compatible via Flash fallback for MP3/MP4 formats. Existing SM2 API seamlessly uses HTML5 where supported, currently experimental; and of course… works on iPad.

Highlights

Experimental HTML5 Audio() support, with Flash fallback for MP3/MP4 as required. (HTML5 disabled by default except for iPad + Palm Pre, but easily configurable.)
100% Flash-free, HTML5-only playback of MP3, MP4 (AAC) and WAV files possible on Apple iPad and Palm Pre (and Safari 4.1.5 on OS X 10.5; buggy behaviour observed with 4.1.5 on OS X 10.6, see https://bugs.webkit.org/show_bug.cgi?id=32159#c9 )
API is unchanged, transparent whether using HTML5 or Flash; SM2 handles switching of technology behind the scenes, depending on browser support.

Here is how it works:

soundManager.useHTML5Audio

Determines whether HTML5 Audio() support is used to play sound, if available, with Flash as the fallback for playing MP3/MP4 (AAC) formats. Browser support for HTML5 Audio varies, and format support (eg. MP3, MP4/AAC, OGG, WAV) can vary by browser/platform.

The SM2 API is effectively transparent, consistent whether using Flash or HTML5 Audio() for sound playback behind the scenes. The HTML5 Audio API is roughly equivalent to the Flash 8 feature set, minus ID3 tag support and a few other items. (Flash 9 features like waveform data etc. are not available.)

SoundManager 2 + useHTML5Audio: Init Process

At DOM ready (if useHTML5Audio = true), a test for Audio() is done followed by a series of canPlayType() tests to see if MP3, MP4, WAV and OGG formats are supported. If none of the “required” formats (MP3 + MP4, by default) are supported natively, then Flash is also added as a requirement for SM2 to start.

soundManager.audioFormats currently defines the list of formats to check (MP3, MP4 and so on), their possible canPlayType() strings (long story short, it’s complicated) and whether or not they are “required” – that is, whether Flash should be loaded if they don’t work under HTML5. (Again, only MP3 + MP4 are supported by Flash.) If you had a page solely using OGG, you could make MP3/MP4 non-required, but many browsers would not play them inline.

SM2 will indicate its state (HTML 5 support or not, using Flash or not) in console.log()-style debug output messages when debugMode = true.

22

Software Engineering / David Humphrey and the hit squad of audio gurus have some new amazing demos

« on: April 20, 2017, 02:44:32 PM »

David Humphrey and the hit squad of audio gurus have some new amazing demos for us. Perfect for a Friday. This is all through the rich Mozilla Audio API work which will hopefully be pushed into other browsers at some point in the not so distant future.
Charles Cliffe has some awesome WebGL visualizations from Audio. David narrates:
What I like most about these (other than the fact that he’s written the music, js libs, and demo) is that these combine a whole bunch of JavaScript libraries: dsp.js, cubicvr.js and beatdetection.js, and processing.js. Some people will tell you that doing anything complex in a browser is going to be slow; but Charles is masterfully proving that you can do many, many things at once and the browser can keep pace.

Corban and Ricard Marxer have been busy exploring how far we can push audio write, and managed to also produce some amazing demos. The first is by Ricard, and is a graphic equalizer (video is here):
The second is by Corban, and shows a JavaScript based audio sampler. His code can loop forward or backward, change playback speed, etc. (video is here)

Chris McCormick has been working on porting Pure Data to JavaScript, and already has some basic components built. Here’s one that combines processing.js and webpd (video is here):
I think that my favourite demo by far this time around is one that I’ve been waiting to see since we first began these experiments. I’ve written in the past that our work could be used to solve many web accessibility problems. A few weeks ago I mentioned on irc that someone should take a shot at building a text to speech engine in JavaScript, now that we have typed arrays. Yury quietly went off and built one based on the flite engine. When you run this, remember that you’re watching a browser speak with no plugins of any kind. This is all done in JavaScript (demo is here, video is here)

In order to do this he had to overcome some interesting problems, for example, how to load large binary voice databases into the page. The straightforward approach of using a JS array was brittle, with JS sometimes running out of stack space trying to initialize the array. After trying various obvious ways, Yury decided to use the web to his advantage, and pushed the binary data into a PNG, then loaded it into a canvas, where getImageData allows him to access the bytes very quickly, using another typed array. The browser takes care of downloading and re-inflating the data automatically.

My favourite line is:
What began as a series of experiments by a small group of strangers, has now turned into something much larger.
What an awesome community you guys have… and we are all benefitting. Thank you.

23

Departments / Mobile Proxies: A New Era Dawns

« on: April 20, 2017, 02:34:05 PM »

This week, Chrome For Android M26 was announced. It has the literally-awesome ability to record video via `getUserMedia()`, but enough about making Skype irrelevant. What’s even more interesting is the new data compression feature. Which, to be clear, is experimental, has to be switched on, doesn’t apply to secure (SSL) sites, and it’s only running in the beta app.

With this feature, Google will be delivering streamlined responses, leading to substantial performance improvements and bandwidth savings. In the latest Mobile Web Thursday’s, Google’s Pete Le Page demonstrated The Verge weighs in at 1.2MB when proxied, down from the hefty 1.9MB it otherwise would have been, and also reports having seen a 65$ reduction elsewhere.
The speed benefit comes in a couple of ways. First, the proxy and the browser communicate via SPDY. Most websites still haven’t enabled SPDY, but Chrome has it built in and Google’s proxy will now act as a SPDY adaptor for any (non-SSL) website. Second, images are returned as WebP format. WebP claims to shave off around 30% and Google claims 60% of bandwidth is images…so that’s about an 18% reduction right there.

Mobile proxying is nothing new for Google. I don’t know the full history, but I recall seeing it as early as 2003 (anyone know?). It would be enabled after a search, and the proxy is still runnning. But this is a proxy for the new era of smartphone browsers, and comes just over a year after Amazon introduced its Silk browser with a similar proxying solution. Unlike the older Opera Mini solution, which provided comfort for resource-challenged feature phones, this new breed of browser is still able to work as a regular browser, but can route through a proxy as needed. What Amazon refers to as a “hybrid browser”.
I have also seen similar capabilities made available as reverse-proxies for site owners. CloudFlare, for example, can compress scripts and optimize images.
Overall, these proxying services will make it easier for developers to deliver a better experience for users. But as they become more popular and better-understood, they will also come with the privacy and security concerns that were ignited soon after Silk browser came out. It will be up to the ecosystem to find the right balance, and Google has so far done well to release this experimentally, and ensuring secure sites go direct to the browser.