Elixir load module file

During debugging Elixir script, one sometimes wants to edit the script file and reload modules defined inside from an Elixir shell. Issue this command from your Elixir shell: iex> import_file("path/to/script_file.exs") Thanks to this blog post!

Customize User Authentication in Grails

In a Grails project, I need to implement an additional authentication mechanism - username plus a short-lived authentication token that the app knows how to decode and verify. The route allows (a 2nd app) login on behalf of a user, without knowing his/her password, as long as there is pre-agreement on secure token generation (This might sound strange, but that's another separate topic, think poorman's single sign on).

Spring Security is complicated, and Grails adds another layer of opacity to it. For this to happen in Grails, we need to

1. create a AuthenticationProvider that defines the customized authentication logic.
2. create a AuthenticationProcessingFilter that defines in what condition such authentication should be activated.

Here are the details:

1. In order to have your own authentication, you need a AuthenticationProvider public class AppUserAuthenticationProvider extends DaoAuthenticationProvider { @Override public boolean supports(Class authentication) { return (AppUserAuthenticationToken.class.isAssignableFrom(authentication)); } @Override protected void additionalAuthenticationChecks(UserDetails userDetails, UsernamePasswordAuthenticationToken authentication) throws AuthenticationException { String name = authentication.getName() String key = (String) authentication.getCredentials() if (key == null) { logger.debug("App Authentication failed: no credentials provided"); throw new BadCredentialsException(messages.getMessage( "AbstractUserDetailsAuthenticationProvider.badCredentials", "Bad credentials"), userDetails); } // customized logic to validate the username and authentication token if (! AppAuthConnector.validateAppKey(name, key)) { logger.debug("App Authentication failed: app_auth_key is invalid"); throw new BadCredentialsException(messages.getMessage( "AbstractUserDetailsAuthenticationProvider.badCredentials", "Bad credentials"), userDetails); } } } public class AppUserAuthenticationToken extends UsernamePasswordAuthenticationToken { public AppUserAuthenticationToken(String username, String key) { super(username, key); } public AppUserAuthenticationToken(UserDetails principal, String credentials) { super(principal, credentials, principal.getAuthorities()); } }

   Note here the customized Provider did not override

   public Authentication authenticate(Authentication authentication) throws AuthenticationException { }

   but rather

   protected void additionalAuthenticationChecks(UserDetails userDetails, UsernamePasswordAuthenticationToken authentication) throws AuthenticationException { }

   The later is one of many steps inside authenticate() method, and authenticate() also includes some default checks that I want to inherit, for example preAuthenticationChecks and postAuthenticationChecks, to check if user account credential has expired, user is locked, disabled, or expired. If we simply override authenticate() without properly implementing these checks, our customized authentication will introduce loopholes, for example allowing an expired user to login from that 2nd app.

2. Register this provider as a bean so it can be used later, in file grails-app/conf/spring/resources.groovy beans = { ... appUserAuthenticationProvider(AppUserAuthenticationProvider) { userDetailsService = ref('userDetailsService') } }
3. Register this provider bean with Spring, in file grails-app/conf/Config.groovy grails.plugin.springsecurity.providerNames = [ 'appUserAuthenticationProvider', 'daoAuthenticationProvider', 'rememberMeAuthenticationProvider’]
4. Now the provider is set, we need to define when to activate this authentication path. In my case I want this authentication mechanism to work for any request, as long as there are two parameters 'app_username', ‘app_key’ in the URL or in request header. We need a AbstractAuthenticationProcessingFilter that can intercept the processing: public class AppUserAuthenticationFilter extends AbstractAuthenticationProcessingFilter { public AppUserAuthenticationFilter() { // the matcher will detect the presence of URL parameters super(new AppUserAuthenticationRequestMatcher()) // what to do after the authentication is successful setAuthenticationSuccessHandler(new AppUserAuthenticationSuccessHandler()) } @Override public Authentication attemptAuthentication(HttpServletRequest req, HttpServletResponse resp) throws AuthenticationException, IOException, ServletException { String key = AppUserAuthenticationRequestMatcher.getKey(req) String username = AppUserAuthenticationRequestMatcher.getName(req) return this.getAuthenticationManager().authenticate(new AppUserAuthenticationToken(username, key)) } } class AppUserAuthenticationSuccessHandler extends SimpleUrlAuthenticationSuccessHandler { @Override public void onAuthenticationSuccess(HttpServletRequest request, HttpServletResponse response, Authentication authentication) throws ServletException, IOException { clearAuthenticationAttributes(request) request.getRequestDispatcher(request.getServletPath()).forward(request, response) } public class AppUserAuthenticationRequestMatcher implements RequestMatcher { public static final String APP_AUTH_NAME = "app_username" public static final String APP_AUTH_KEY = "app_key" public boolean matches(javax.servlet.http.HttpServletRequest request) { String uri = request.getRequestURI() if(uri != null && AppUserAuthenticationRequestMatcher.getKey(request) != null && AppUserAuthenticationRequestMatcher.getName(request) != null) { return true; } else { return false; } } public static String getRequestParam(HttpServletRequest request, String param) { String k = request.getParameter(param); if(k == null) { String hn = "X-" + param; k = request.getHeader(hn); } return k; } public static String getKey(HttpServletRequest request) { return AppUserAuthenticationRequestMatcher.getRequestParam(request, AppUserAuthenticationRequestMatcher.APP_AUTH_KEY); } public static String getName(HttpServletRequest request) { return AppUserAuthenticationRequestMatcher.getRequestParam(request, AppUserAuthenticationRequestMatcher.APP_AUTH_NAME); } }
5. Register this filter as a bean, in file grails-app/conf/spring/resources.groovy beans = { ... appUserAuthenticationFilter(AppUserAuthenticationFilter) { sessionAuthenticationStrategy = ref('sessionAuthenticationStrategy') authenticationManager = ref('authenticationManager') authenticationFailureHandler = ref('authenticationFailureHandler') rememberMeServices = ref('rememberMeServices') authenticationDetailsSource = ref('authenticationDetailsSource') } }
6. activate the filter during the bootstrap, in grails-app/conf/BootStrap.groovy class BootStrap { def init = { servletContext -> ... SpringSecurityUtils.clientRegisterFilter('appUserAuthenticationFilter', SecurityFilterPosition.SECURITY_CONTEXT_FILTER.order + 10) }

That should do it. Enjoy!

Even with Grails' "Convention over Configuration" paradigm, aimed at simplify the coding, there is still a high learning curve to do get in the groove of it, especially if you want to something different from the default. The layers are thick, and the magic is ever more mysterious. The root cause is the enormous amount of flexibility the framework is trying to offer - every little decision is wrapped up and ready to be swapped out.

Reference: Grails Custom AuthenticationProvider by Kali Kallin.

<? super T> and <? extends T> in Java

Today, I came across some Java code from a Java library we used, as with many framework/library code bases, it contains lots of Generics, Collections, and Type Wildcards (?). For example: <? super T> and <? extends T> or public static <T extends Comparable<? super T>> void sort(List<T> list) The syntax is complex, and it prompted me to study further the wildcard of Generic types. Thanks to Google and Stack Overflow, the online material is good.

• Difference between <? super T> and <? extends T> in Java
• What is PECS (Producer Extends Consumer Super)?
• Bounded Wildcards Increase Applicability

In my own words, which is less precise than in these articles, but maybe easier to understand:

• If you need to only read through a data structure, declare it with extends, as it can guarantee the object type, and you will be type safe when using the objects.
• If you need a data structure that allow to put objects into it, then declare it with super, as it allows more type flexibility.

This code example helps show both cases (copied from Generics FAQ):

public class Collections { public static <T> void copy ( List<? super T> dest, List<? extends T> src) { // bounded wildcard parameterized types for (int i=0; i<src.size(); i++) dest.set(i,src.get(i)); } }

As you can see, the src list is declared as extends, since you only need to read from it (get), and the dest list is declared as super, as you need to write to it (set).

Enjoy!

Disconnect a PHP PDO connection to RDBMS

PHP's PDO connection management is a bit arcade. It does not offer an explicit way to disconnect from database. Here is the PHP's documentation:

http://php.net/manual/en/pdo.connections.php

Inside which it will tell you to set the variable that holds the PDO connection object to null. However the catch is you also need to set all other objects which might hold reference to the PDO object to null as well, and it can be cumbersome as expressed in the comments section of that page. Here is an example: $dbh = new PDO('mysql:host=localhost;dbname=test', $user, $pass); // use the connection here, note the side effect of // a new $sth reference to the PDO $sth = $dbh->query('SELECT * FROM foo'); // and now we're done; close both $sth and $dbh ! $sth = null; // this is not obvious $dbh = null; Enjoy!

dos2unix

The text file I got from a colleague is full of ^M and crashes my parser running on Mac and Linux. This is due to the very unfortunate colleague is still stuck with a Windows machine, and the line break is coded differently on these systems.

I tried to download the dos2unix util for Mac and installed it smoothly, however it does not work for some reason, the ^M's are still there after running it. Bummer! Short of time, I have to roll my own solution. It is yet another one-liner:

$ cat dos2unix #!/bin/bash sed -i 's/^M/\n/g' $1

However the main challenge is to type in ^M character correctly inside the script. It is NOT a literal ^ and a literal M, but rather a single character.

• In vi, you type: ctrl-v then ctrl-m
• In emacs, you type: ctrl-q then ctrl-m

ctrl-v means you hold down the ctrl key and v key at the same time.

Enjoy!

[Update 2016-12-21]:

Coming back to the problem, upon further inspection on the problematic file, using $ od -c filename It turns out the line break is only represented as \r, and I guess that's why the regular dos2unix does not work, which expects \r\n. Knowing this, another fix come to mind (without having to deal with typing ^M character): $ sed -i -e 's/\r/\n/g' filename Also my docker infested colleague suggested another general solution for running standard dos2unix when it works, without having to download and install the dos2unix in the hosting environment: $ docker run --rm -it -v `pwd`:/data/ alpine dos2unix /data/filename

This assumes the filename is in the `pwd`, and you have docker installed on the hosting environment. This basically launch a minimal docker machine and use the dos2unix that comes with it. However it won't solve my specific file with non traditional line break.

Enjoy!

Find out the Perl modules installed and their versions

It is actually just a one-liner:
$ whichpm.sh module::name perl_bin_path It prints out all the Perl modules accessible by the particular Perl interpreter, and then the version of the given module.
$ cat whichpm.sh #!/bin/bash echo 'print map { sprintf( "%20s : %s\n", $_, $INC{$_} ) } sort keys %INC; print "\n '$1' version : $'$1'::VERSION\n\n"' | $2 "-M$1" $ ./whichpm.sh LWP::UserAgent /usr/bin/perl Carp.pm : /System/Library/Perl/5.18/Carp.pm LWP.pm : /Library/Perl/5.18/LWP.pm LWP/UserAgent.pm : /Library/Perl/5.18/LWP/UserAgent.pm Storable.pm : /System/Library/Perl/5.18/darwin-thread-multi-2level/Storable.pm Time/Local.pm : /System/Library/Perl/5.18/Time/Local.pm URI.pm : /System/Library/Perl/Extras/5.18/URI.pm strict.pm : /System/Library/Perl/5.18/strict.pm vars.pm : /System/Library/Perl/5.18/vars.pm warnings.pm : /System/Library/Perl/5.18/warnings.pm LWP::UserAgent version : 6.13

The reason for the 2nd argument (perl interpreter) is that there could be multiple Perl interpreters on a system and they tend to have different version and also the supporting modules.

Of course, to make it a reusable script, you will have to add help message, input argument checking, etc.

Enjoy!

Batch inserts to SQLite

I have a need to insert hundreds of thousands of rows to a SQLite database. Initially I tried to insert one row at a time, but the speed was very slow, even with a prepared statement. Using the multiple-rows insert statement significantly increased the performance.

Here is the PHP code sample:

Option 1 (Slow) $stmt = $dbh->prepare("insert into wordindex (key, value) values (:k, :v)"); foreach ($data as $k => $v) { $stmt->execute(array(':k' => $k, ':v' => $v)); } Option 2 (Fast, up to 100 times)

uses the SQL syntax like the following : insert into table (col1, col2) value (a1, a2), (b1, b2), (c1, c2), ... Here is what the PHP code looks like: $batchSize = 500; $chunks = array_chunk($data, $batchSize, TRUE); foreach($chunks as $chunk) { $sql = "insert into wordindex (key, value) values "; $pairs = array(); foreach($chunk as $k => $v) { $pairs[] = "('$k','$v')"; } $sql .= implode(',',$pairs); $dbh->query($sql); } Since we are not using SQL parameter binding in option 2, we need to pay attention to values for $k and $v, to avoid SQL injection.

Enjoy!