joubin's Kardz


Executing Dynamic SQL

Use EXEC to execute a dynamically generated SQL statement or query.

Executes a command string or character string within a Transact-SQL batch, or one of the following modules: system stored procedure, user-defined stored procedure, CLR stored procedure, scalar-valued user-defined function, or extended stored procedure.

declare @sourceDb sysname = N'[dbSource].[schema].[SourceTable]';
declare @sqlCommand varchar(max);

-- get count of source database table
SET @sqlCommand = 'select count(*) from ' + @sourceDb;
EXEC (@sqlCommand)

References:

Shared by joubin on Mon Oct 22 2018

Temp table, TSQL

  • Syntax: Create Table #UseTempTable
  • Temporary tables are created by prefixing your table name with a #.
  • To get a list of temp tables: select * from tempdb..sysobjects;

References:





Shared by joubin on Thu Oct 18 2018

Pivot tables, Excel

A pivot table allows for reorganizing and summarizing selected columns and rows in a data set. This data set could be from a spreadsheet or a database table to form a desired report.

A pivot table doesn't change the underlying data.

Pivoting means to turn a data set to view it from different perspectives.

References:

Shared by joubin on Tue Oct 16 2018

Tuckman's stages of group development

The forming–storming–norming–performing model of group development was first proposed by Bruce Tuckman in 1965.

https://en.wikipedia.org/wiki/Tuckman%27s_stages_of_group_development







Shared by joubin on Thu Oct 11 2018

Http Status Code - Redirects

  • 302 - temporary redirect
  • 301 - permanent redirect

References:

Shared by joubin on Thu Oct 18 2018

Microsoft Graph Security API

To build intelligent security solutions correlate alerts, get additional context to speed up investigation, automate security operations, use Microsoft Graph Security API.

  • A federated system
  • Microsoft Graph Security API
  • Security API notification subscription
  • Secure Score, system wide
  • Identity security score, AD

https://www.microsoft.com/en-us/security/intelligence-security-api





Shared by joubin on Thu Sep 27 2018

MS Ignite, Migrate to Cosmos DB

  • GA of Cassandra API
    • Table API
    • MongoDB
  • Keyspace ?
  • CQL Query Builder
  • CQL Shell (CQLSH)
  • Cassandra Stress, to verify performance
  • Scale throughput
    • Change the throughput for any table
  • SDK access
    • org.apache.spark...

Ignite 2018 session: https://myignite.techcommunity.microsoft.com/sessions/65369?source=schedule#ignite-html-anchor

Cheatsheets: https://docs.microsoft.com/en-us/azure/cosmos-db/query-cheat-sheet







Shared by joubin on Wed Sep 26 2018


NSAT

Net customer/user Satisfaction (NSAT)

NSAT is an index of net satisfaction scores across a number of segments and audiences as measured by the Global Relationship Study.

Shared by joubin on Tue Oct 02 2018

MS Ignite, Kubernetes on Azure

  • Gabe Monroy
  • Kubernetes, container orchestration
  • Portability
  • Extensible
  • Self healing
  • Kubectl
  • Etcd
  • Kubelet, heart beat back to the API server
  • Ingress
  • Kube-proxy
  • Lift and Shift application moving strategy, without refactoring your app
  • Azure managed control plane
    • Devops project, Azure resource
    • Azure Dev Spaces ?
    • Helm chart ?
    • CI/CD pipelines
    • Key Vault is used to store secrets for Kubernetes
    • Vnet ?
    • Traffic Manager, in Azure
    • ACI ?, Azure Container Instances
  • Azure Monitor for containers
  • RBAC control for fine grained Kubernetes resources access control
  • Scenarios
    • Lift and Shift to containers
    • Micro services
    • Machine Learning scenarios
    • IoT
      • Azure IoT Edge
  • Maersk
    • NSAT, customer satisfaction
  • Terraform
  • CaaS
  • OpenAI, a not for profit organization
  • Azure IoT Edge






















Shared by joubin on Tue Sep 25 2018

Onnx, Deep Learning

Open Neural Network Exchange is a format for deep learning models that allows interoperability between different open source AI frameworks.

ONNX is being co-developed as an open-source project by:

  • Microsoft
  • Amazon
  • Facebook

https://en.wikipedia.org/wiki/Onnx

Shared by joubin on Sun Sep 30 2018

MS Ignite, Event driven apps with CosmosDB

  • Event Notification pattern
  • Tight coupling prevention in the design
  • Eg adding a queue between the order module and the shipping module
  • System becomes more resilient
  • Events should have as much context around then as they can
  • Event sourcing pattern, writing the events to a log
    • Audit History
    • Decoupling
    • Materialized views
    • Integration with other systems
    • Teleporting ?
  • Microservices
  • Fault tolerance
  • Functions can be triggered based on an event
  • Azure CosmosDB
  • RU: Request Unit
  • Five consistency models ?
  • A multi model database, key value, column family, documents, graph
  • ARS, Atomic Record Sequence
  • Schema agnostic
  • Change Feed: log updates or others
    • Triggers for Azure Functions
    • Stream Processing
    • Data Movement, moving the data downstream
  • Multiple consumers of the change feed
  • Azure Functions
    • Bindings and Triggers
    • Proxies
    • Monitoring
    • Local Debugging
    • CI/CD
    • Doable Functions, will save their own states
  • Azure Event Grid
    • Event handlers: Functions, etc
  • Use the Change Feed to light up your micro services architecture
  • Jet.com uses micro services on CosmosDB and Azure Event Grid
  • Skype as well
  • CQRS
























Shared by joubin on Tue Sep 25 2018

Real-time Messaging with Azure SignalR Service

  • Roadmap
    • Q2, had public preview
    • Q3, GA
    • Q4
  • Scale with backplane with Redis etc
  • Azure Function binding, in preview
  • For GA, Free Tier and standard
  • Auto Scaling... ??
  • Scenarios
    • In app notifications
    • Live dashboard
    • Co authoring
  • It's realtime messaging
  • An Azure resource
  • dotnet build; dotnet run;
  • Azure SignalR Service on asp.net Core SignalR
  • Hub base class
  • First negotiates. If configured to use Azure SignalR service, it returns negotiation points on Azure.
  • Rest API
    • Gyro tracker demo
  • Azure Function binding for SignalR Service


































Shared by joubin on Thu Sep 27 2018

Node.js apps on Azure with VS Code

  • 12 Factor
  • Mongoose SDK Node.js sdk
  • Develop locally with MongoDB
  • Switch to CosmosDB in the cloud
    • Choose MongoDB api for CosmosDB
  • Make app cloud ready
  • dotenv package ??
  • VS Code, Ctrl+shift B, run commands










Shared by joubin on Thu Sep 27 2018

LiveTiles, QnA Maker

  • LiveTiles Bots
  • Scenarios
    • Personal Assistant
    • Team Assistant
    • Enterprise Assistant
    • External Assistant
  • What's a typical development cycle?
  • Site application as a service as a bot!!
  • QnA Maker
  • LiveTiles Bot designer
  • AIconics ?
  • EthicalOS.org
























Shared by joubin on Thu Sep 27 2018

Top Tips for Git Powered Teams

  • git push origin master:demo
    • If you forget to create a branch before pushing code remotely
  • Branch policies
    • Require PR
    • Item linking
  • Build Validation policy
    • Build must pass before merge into branch/master
  • Checkout pull requests locally ??
  • Continuous Code Quality with Automation
    • Static code analysis
      • sonarqube/sonarcloud.io
      • whitesource, security scanning
    • Drop these into pipeline
  • Git Large File Support
    • Git LFS
    • Instead of shallow cloning
    • BFG Repo Cleaner ??
  • Always use the latest version of Git
    • git update
  • Trunk-based development
    • GitHub flow is different that Release flow branching structure
    • master is never in production?? Why?
    • Cherry pick a Pull request into a new branch






















Shared by joubin on Thu Sep 27 2018

MS Ignite, Azure Pipelines and GitHub

  • Extensions
    • Slack, SonarCloud
    • Over 500 extensions
  • Integration with GitHub
  • Start with GitHub market place
    • Go to continuous integration
    • Setup Azure pipelines
    • Will land in Azure DevOps
    • Example: GitHub/microsoft/vscode
      • Has badges for Azure Pipelines
  • CredScan ?, A step that check for credentials as part of a commit


















Shared by joubin on Wed Sep 26 2018

MS Ignite, Azure Relay, Hubs, Grid, Service Bus

  • Azure Relay
    • Connect Systems
  • Event Hubs
  • Event Grid
  • Messages
    • Fact
      • Time Series, logs, streams
        • Event Hubs for streaming data
      • Discrete
        • Not time series i.e. no order to the messages
        • Event Grid, event based programming
    • Command
      • Service Bus, Enterprise messaging
  • Service Bus Explorer
  • System to system messaging
    • Contrast this with
      • Notification hub
      • SignalR hub








Shared by joubin on Wed Sep 26 2018

MS Ignite, Automated ML & Hyperparameter Tuning

  • Labeling process
  • Hyperparameters??
  • scikit learn algorithm cheat sheet
  • Automated machine learning
    • To give you a model faster
    • Submitting experiments to Automated ML
  • Jupyter notebook
  • Cross validation ??
  • KNeighbors ?
  • Set metrics, e.g. accuracy, AUC etc
  • Confusion matrix, TP TN etc
  • Configure Automated ML by setting metrics

Benefits of Automated ML

  • Model creation time, from days to hours
  • Robust benchmarking process for ML projects
  • Enable domain experts to leverage ML
  • Is an excellent starting point for selection of algorithm
  • Currently: Classification/binary, and regression

Power BI

  • Automated ML is baked into Power BI

Hyperparameter tuning

  • Adjustable parameters to given model training
  • Hyperparameter configuration
  • Find the configuration that result in best performance
  • Best accuracy
  • Number of layers in deep learning
    • This is in addition to the training datasets
  • Automated hyperparameter tuning in Azure ML
  • Early terminations of poorly running models
  • Bandit policy ??
  • This capability is available for free, for now!






























Shared by joubin on Wed Sep 26 2018

MS Ignite, Common Data Services for Apps

  • CDS, Common Data Services
  • An Azure based service
  • PowerApps portal
    • Define custom entities
    • Relationships, Lookups
    • Optionsets eg enums
    • Calculated Fields, computed columns
    • Roll-up Fields, aggregates
  • Microsoft Flows?

https://docs.microsoft.com/en-us/powerapps/maker/common-data-service/data-platform-intro

Shared by joubin on Wed Sep 26 2018